Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
On Tue, Nov 25, 2014 at 9:35 PM, Nidhi nidhi.ashwani.aggar...@gmail.com wrote: Thank you Patrik, we are restricting the sender to send lesser queries now. Sounds good. Could you tell us what exactly is to be used from the fix ? We have akka 2.3.6. There is nothing you need to enable to use the improvements, apart from using Akka 2.3.3 or later. Latest is 2.3.7 (always recommended to use latest stable). Details can be found in the pull request, if you are interested. https://github.com/akka/akka/pull/2116 I think I published some benchmark results here: https://groups.google.com/forum/#!msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J Note that serialization is often a bottleneck. In case you use (default) Java serialization you should use something faster. Let me know if I misunderstood your question. Regards, Patrik On Tuesday, November 25, 2014 2:46:44 AM UTC-5, Patrik Nordwall wrote: On Mon, Nov 24, 2014 at 1:32 AM, Nidhi nidhi.ashwa...@gmail.com wrote: Hello all, we are working on a course project to simulate twitter server and twitter users and test the server for the load it can handle. We have users on one system (with one client master and around 10,000 user actors) and server on another machine(one master and 1000 worker actors) to resolve the queries it gets from the user actors remotely. We are sending an average of 6000 queries/sec to the server. We have one TCP connection between the 2 systems and we get the following warning [TwitterServerSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp://TwitterServerSystem@122.122.122.122:2552/system/ endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FTwit terClientSystem%40127.0.0.1%3A56833-0/endpointWriter http://www.google.com/url?q=http%3A%2F%2FTwitterServerSystem%4010.136.43.158%3A52335%2Fsystem%2FendpointManager%2FreliableEndpointWriter-akka.tcp%253A%252F%252FTwitterClientSystem%2540127.0.0.1%253A56833-0%2FendpointWritersa=Dsntz=1usg=AFQjCNFv8cFbjN-nkft3bH--gLuNWbbdVQ] [65138] buffered messages in EndpointWriter for [akka.tcp:// TwitterClientSystem@127.0.0.1:56833 http://www.google.com/url?q=http%3A%2F%2FTwitterClientSystem%40127.0.0.1%3A56833sa=Dsntz=1usg=AFQjCNHDGGNE6uGCLY9WRq5G-oSxo0FaGQ]. You should probably implement flow control to avoid flooding the remote connection. This warning indicates that many remote messages have been queued on the sender side. It is configured by: # Log warning if the number of messages in the backoff buffer in the endpoint # writer exceeds this limit. It can be disabled by setting the value to off. akka.remote.log-buffer-size-exceeding = 5 The recommendation about flow control means that you should add some application level protocol (messages) between sender and receiver that controls how much the sender is allowed to produce before it stops sending more. Without that you will get out of memory if the sender continues to produce faster than what can be consumed. Just came across this thread and thought it is relevant to our problem. We know since we have one tcp connection(can we increase the number of connections?), it might be a bottleneck. Both the server and client machines buffer messages for each other. How do we go about using this new fix ? The improvements for sending bursts of remote messages have been included in Akka 2.3.x since 2.3.3. Regards, Patrik Thank you. Regards, Nidhi On Friday, April 25, 2014 9:48:55 AM UTC-4, Patrik Nordwall wrote: Boris, you should try the timestamped snapshot 2.3-20140425-151510 that is published to repo http://repo.akka.io/snapshots/ It is supposed to handle bursts of many messages without (much) degraded throughput or false failure detection. More details here: https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J Regards, Patrik On Mon, Mar 24, 2014 at 5:00 PM, √iktor Ҡlang viktor...@gmail.com wrote: Nice! On Mon, Mar 24, 2014 at 4:55 PM, Boris Capitanu bor...@gmail.com wrote: Anyway, one thing to try is to set akka.remote.backoff-interval to a larger value while setting the send-buffer-size to 1024000b. I would try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a good way to test our hypothesis. I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and I do see the timings becoming more consistent (albeit worse). The standard deviation of the timings observed is much lower. Well - I think we narrowed down the issue. I'll wait for a fix... I'll be glad to test any nightly builds that include a fix if it would be helpful. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/c urrent/additional/faq.html Search the archives: https://groups.google.com/grou p/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Thank you Patrik, we are restricting the sender to send lesser queries now. Could you tell us what exactly is to be used from the fix ? We have akka 2.3.6. On Tuesday, November 25, 2014 2:46:44 AM UTC-5, Patrik Nordwall wrote: On Mon, Nov 24, 2014 at 1:32 AM, Nidhi nidhi.ashwa...@gmail.com javascript: wrote: Hello all, we are working on a course project to simulate twitter server and twitter users and test the server for the load it can handle. We have users on one system (with one client master and around 10,000 user actors) and server on another machine(one master and 1000 worker actors) to resolve the queries it gets from the user actors remotely. We are sending an average of 6000 queries/sec to the server. We have one TCP connection between the 2 systems and we get the following warning [TwitterServerSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp://TwitterServerSystem@122.122.122.122:2552/system/ endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% 2FTwitterClientSystem%40127.0.0.1%3A56833-0/endpointWriter http://www.google.com/url?q=http%3A%2F%2FTwitterServerSystem%4010.136.43.158%3A52335%2Fsystem%2FendpointManager%2FreliableEndpointWriter-akka.tcp%253A%252F%252FTwitterClientSystem%2540127.0.0.1%253A56833-0%2FendpointWritersa=Dsntz=1usg=AFQjCNFv8cFbjN-nkft3bH--gLuNWbbdVQ] [65138] buffered messages in EndpointWriter for [akka.tcp:// TwitterClientSystem@127.0.0.1:56833 http://www.google.com/url?q=http%3A%2F%2FTwitterClientSystem%40127.0.0.1%3A56833sa=Dsntz=1usg=AFQjCNHDGGNE6uGCLY9WRq5G-oSxo0FaGQ]. You should probably implement flow control to avoid flooding the remote connection. This warning indicates that many remote messages have been queued on the sender side. It is configured by: # Log warning if the number of messages in the backoff buffer in the endpoint # writer exceeds this limit. It can be disabled by setting the value to off. akka.remote.log-buffer-size-exceeding = 5 The recommendation about flow control means that you should add some application level protocol (messages) between sender and receiver that controls how much the sender is allowed to produce before it stops sending more. Without that you will get out of memory if the sender continues to produce faster than what can be consumed. Just came across this thread and thought it is relevant to our problem. We know since we have one tcp connection(can we increase the number of connections?), it might be a bottleneck. Both the server and client machines buffer messages for each other. How do we go about using this new fix ? The improvements for sending bursts of remote messages have been included in Akka 2.3.x since 2.3.3. Regards, Patrik Thank you. Regards, Nidhi On Friday, April 25, 2014 9:48:55 AM UTC-4, Patrik Nordwall wrote: Boris, you should try the timestamped snapshot 2.3-20140425-151510 that is published to repo http://repo.akka.io/snapshots/ It is supposed to handle bursts of many messages without (much) degraded throughput or false failure detection. More details here: https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J Regards, Patrik On Mon, Mar 24, 2014 at 5:00 PM, √iktor Ҡlang viktor...@gmail.com wrote: Nice! On Mon, Mar 24, 2014 at 4:55 PM, Boris Capitanu bor...@gmail.com wrote: Anyway, one thing to try is to set akka.remote.backoff-interval to a larger value while setting the send-buffer-size to 1024000b. I would try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a good way to test our hypothesis. I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and I do see the timings becoming more consistent (albeit worse). The standard deviation of the timings observed is much lower. Well - I think we narrowed down the issue. I'll wait for a fix... I'll be glad to test any nightly builds that include a fix if it would be helpful. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/ current/additional/faq.html Search the archives: https://groups.google.com/ group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com. To post to this group, send email to akka...@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ * ——— **Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/ current/additional/faq.html Search the archives: https://groups.google.com/ group/akka-user --- You received this message because you are subscribed
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hello all, we are working on a course project to simulate twitter server and twitter users and test the server for the load it can handle. We have users on one system (with one client master and around 10,000 user actors) and server on another machine(one master and 1000 worker actors) to resolve the queries it gets from the user actors remotely. We are sending an average of 6000 queries/sec to the server. We have one TCP connection between the 2 systems and we get the following warning [TwitterServerSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp:// TwitterServerSystem@122.122.122.122:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FTwitterClientSystem%40127.0.0.1%3A56833-0/endpointWriter http://www.google.com/url?q=http%3A%2F%2FTwitterServerSystem%4010.136.43.158%3A52335%2Fsystem%2FendpointManager%2FreliableEndpointWriter-akka.tcp%253A%252F%252FTwitterClientSystem%2540127.0.0.1%253A56833-0%2FendpointWritersa=Dsntz=1usg=AFQjCNFv8cFbjN-nkft3bH--gLuNWbbdVQ] [65138] buffered messages in EndpointWriter for [akka.tcp:// TwitterClientSystem@127.0.0.1:56833 http://www.google.com/url?q=http%3A%2F%2FTwitterClientSystem%40127.0.0.1%3A56833sa=Dsntz=1usg=AFQjCNHDGGNE6uGCLY9WRq5G-oSxo0FaGQ]. You should probably implement flow control to avoid flooding the remote connection. Just came across this thread and thought it is relevant to our problem. We know since we have one tcp connection(can we increase the number of connections?), it might be a bottleneck. Both the server and client machines buffer messages for each other. How do we go about using this new fix ? Thank you. Regards, Nidhi On Friday, April 25, 2014 9:48:55 AM UTC-4, Patrik Nordwall wrote: Boris, you should try the timestamped snapshot 2.3-20140425-151510 that is published to repo http://repo.akka.io/snapshots/ It is supposed to handle bursts of many messages without (much) degraded throughput or false failure detection. More details here: https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J Regards, Patrik On Mon, Mar 24, 2014 at 5:00 PM, √iktor Ҡlang viktor...@gmail.com javascript: wrote: Nice! On Mon, Mar 24, 2014 at 4:55 PM, Boris Capitanu bor...@gmail.com javascript: wrote: Anyway, one thing to try is to set akka.remote.backoff-interval to a larger value while setting the send-buffer-size to 1024000b. I would try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a good way to test our hypothesis. I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and I do see the timings becoming more consistent (albeit worse). The standard deviation of the timings observed is much lower. Well - I think we narrowed down the issue. I'll wait for a fix... I'll be glad to test any nightly builds that include a fix if it would be helpful. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com javascript:. To post to this group, send email to akka...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ * ——— **Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com javascript:. To post to this group, send email to akka...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe http://typesafe.com/ - Reactive apps on the JVM Twitter: @patriknw JOIN US. REGISTER TODAY! http://www.scaladays.org/ Scala http://www.scaladays.org/ Days http://www.scaladays.org/ June 16th-18th, http://www.scaladays.org/ Berlin http://www.scaladays.org/ -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
On Mon, Nov 24, 2014 at 1:32 AM, Nidhi nidhi.ashwani.aggar...@gmail.com wrote: Hello all, we are working on a course project to simulate twitter server and twitter users and test the server for the load it can handle. We have users on one system (with one client master and around 10,000 user actors) and server on another machine(one master and 1000 worker actors) to resolve the queries it gets from the user actors remotely. We are sending an average of 6000 queries/sec to the server. We have one TCP connection between the 2 systems and we get the following warning [TwitterServerSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp:// TwitterServerSystem@122.122.122.122:2552/system/endpointManager/ reliableEndpointWriter-akka.tcp%3A%2F%2FTwitterClientSystem%40127.0. 0.1%3A56833-0/endpointWriter http://www.google.com/url?q=http%3A%2F%2FTwitterServerSystem%4010.136.43.158%3A52335%2Fsystem%2FendpointManager%2FreliableEndpointWriter-akka.tcp%253A%252F%252FTwitterClientSystem%2540127.0.0.1%253A56833-0%2FendpointWritersa=Dsntz=1usg=AFQjCNFv8cFbjN-nkft3bH--gLuNWbbdVQ] [65138] buffered messages in EndpointWriter for [akka.tcp:// TwitterClientSystem@127.0.0.1:56833 http://www.google.com/url?q=http%3A%2F%2FTwitterClientSystem%40127.0.0.1%3A56833sa=Dsntz=1usg=AFQjCNHDGGNE6uGCLY9WRq5G-oSxo0FaGQ]. You should probably implement flow control to avoid flooding the remote connection. This warning indicates that many remote messages have been queued on the sender side. It is configured by: # Log warning if the number of messages in the backoff buffer in the endpoint # writer exceeds this limit. It can be disabled by setting the value to off. akka.remote.log-buffer-size-exceeding = 5 The recommendation about flow control means that you should add some application level protocol (messages) between sender and receiver that controls how much the sender is allowed to produce before it stops sending more. Without that you will get out of memory if the sender continues to produce faster than what can be consumed. Just came across this thread and thought it is relevant to our problem. We know since we have one tcp connection(can we increase the number of connections?), it might be a bottleneck. Both the server and client machines buffer messages for each other. How do we go about using this new fix ? The improvements for sending bursts of remote messages have been included in Akka 2.3.x since 2.3.3. Regards, Patrik Thank you. Regards, Nidhi On Friday, April 25, 2014 9:48:55 AM UTC-4, Patrik Nordwall wrote: Boris, you should try the timestamped snapshot 2.3-20140425-151510 that is published to repo http://repo.akka.io/snapshots/ It is supposed to handle bursts of many messages without (much) degraded throughput or false failure detection. More details here: https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J Regards, Patrik On Mon, Mar 24, 2014 at 5:00 PM, √iktor Ҡlang viktor...@gmail.com wrote: Nice! On Mon, Mar 24, 2014 at 4:55 PM, Boris Capitanu bor...@gmail.com wrote: Anyway, one thing to try is to set akka.remote.backoff-interval to a larger value while setting the send-buffer-size to 1024000b. I would try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a good way to test our hypothesis. I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and I do see the timings becoming more consistent (albeit worse). The standard deviation of the timings observed is much lower. Well - I think we narrowed down the issue. I'll wait for a fix... I'll be glad to test any nightly builds that include a fix if it would be helpful. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/ current/additional/faq.html Search the archives: https://groups.google.com/ group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com. To post to this group, send email to akka...@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ * ——— **Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/ current/additional/faq.html Search the archives: https://groups.google.com/ group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com. To post to this group, send email to akka...@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Boris, you should try the timestamped snapshot 2.3-20140425-151510 that is published to repo http://repo.akka.io/snapshots/ It is supposed to handle bursts of many messages without (much) degraded throughput or false failure detection. More details here: https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J Regards, Patrik On Mon, Mar 24, 2014 at 5:00 PM, √iktor Ҡlang viktor.kl...@gmail.comwrote: Nice! On Mon, Mar 24, 2014 at 4:55 PM, Boris Capitanu bor...@gmail.com wrote: Anyway, one thing to try is to set akka.remote.backoff-interval to a larger value while setting the send-buffer-size to 1024000b. I would try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a good way to test our hypothesis. I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and I do see the timings becoming more consistent (albeit worse). The standard deviation of the timings observed is much lower. Well - I think we narrowed down the issue. I'll wait for a fix... I'll be glad to test any nightly builds that include a fix if it would be helpful. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ * ——— **Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe http://typesafe.com/ - Reactive apps on the JVM Twitter: @patriknw JOIN US. REGISTER TODAY! http://www.scaladays.org/ Scala http://www.scaladays.org/ Days http://www.scaladays.org/ June 16th-18th, http://www.scaladays.org/ Berlin http://www.scaladays.org/ -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Boris, Thanks for packing it in a reproducible sample. As Viktor points out the Java serializer is one of the prime suspect for bottlenecks in remote communication. Here you have found something that I think can be improved in Akka remoting. I can see that major time is consumed in EndpointWriter stashing and unstashing messages. That happens when the transport can't write. Perhaps some internal buffer is full. Please create a tickethttp://doc.akka.io/docs/akka/current/project/issue-tracking.htmland we will look into if we can improve the efficiency of this buffering mechanism. What you should do is to implement flow control in the application layer to avoid producing faster than what can be consumed. and also, implement a good serializer for your messages. /Patrik On Mon, Mar 24, 2014 at 1:58 AM, √iktor Ҡlang viktor.kl...@gmail.comwrote: Hello Boris, Then you are likely to benchmark the slowness of Java Serialization. To maximize performance, one needs to choose an appropriate serializer. On Mon, Mar 24, 2014 at 1:51 AM, Boris Capitanu bor...@gmail.com wrote: Hello Viktor, I am using whatever default serializer Akka comes with. I have not configured anything special for that. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ * ——— **Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe http://typesafe.com/ - Reactive apps on the JVM Twitter: @patriknw -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Then you are likely to benchmark the slowness of Java Serialization. Viktor, thank you for your feedback. I'm not convinced the serializer is at fault with this. Regardless of how inefficient the serializer is, it should not take a variable amount of time to serialize identical objects. For example, when the sender is told to send 100,000 messages to receiver: 16:07:33.976 [INFO] - delta = 475 ms, received = 40,000 16:07:34.585 [INFO] - delta = 610 ms, received = 60,000 16:07:35.065 [INFO] - delta = 479 ms, received = 80,000 a log message is generated once for each 20K messages the receiver receives; the delta value is the time elapsed since last log message was printed (so it shows how long it took to receive the next 20K messages); given that all the messages are of the same size (as in case class Message(id :Int)), I would expect the time it takes for the receiver to receive 20K messages should be more or less constant; compare the above with: 17:46:44.280 [INFO] - delta = 14,290 ms, received = 40,000 17:47:01.804 [INFO] - delta = 17,524 ms, received = 60,000 17:47:19.130 [INFO] - delta = 17,325 ms, received = 80,000 17:47:35.865 [INFO] - delta = 16,736 ms, received = 100,000 which is output generated in a different run, when the sender was told to send 1,600,000 messages to the receiver. In the former case each 25K set of messages is received in ~600ms... while in the latter case, over 16 seconds. I'm not seeing how the difference can be explained by the serializer used. Even if a single message is serialized in a long time, it should be the same amount of long time regardless of how many messages the sender is told to send to receiver. Am I missing something? Thank you! -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Neither networked nor multithreaded applications have constant performance. Thread scheduling artifacts, cache effects, timing issues in handoffs, network congestion, retransmissions etc. My suggestion is to take out the known slow parts first. Then measure and tune. On Mar 24, 2014 8:31 AM, Boris Capitanu bor...@gmail.com wrote: Then you are likely to benchmark the slowness of Java Serialization. Viktor, thank you for your feedback. I'm not convinced the serializer is at fault with this. Regardless of how inefficient the serializer is, it should not take a variable amount of time to serialize identical objects. For example, when the sender is told to send 100,000 messages to receiver: 16:07:33.976 [INFO] - delta = 475 ms, received = 40,000 16:07:34.585 [INFO] - delta = 610 ms, received = 60,000 16:07:35.065 [INFO] - delta = 479 ms, received = 80,000 a log message is generated once for each 20K messages the receiver receives; the delta value is the time elapsed since last log message was printed (so it shows how long it took to receive the next 20K messages); given that all the messages are of the same size (as in case class Message(id :Int)), I would expect the time it takes for the receiver to receive 20K messages should be more or less constant; compare the above with: 17:46:44.280 [INFO] - delta = 14,290 ms, received = 40,000 17:47:01.804 [INFO] - delta = 17,524 ms, received = 60,000 17:47:19.130 [INFO] - delta = 17,325 ms, received = 80,000 17:47:35.865 [INFO] - delta = 16,736 ms, received = 100,000 which is output generated in a different run, when the sender was told to send 1,600,000 messages to the receiver. In the former case each 25K set of messages is received in ~600ms... while in the latter case, over 16 seconds. I'm not seeing how the difference can be explained by the serializer used. Even if a single message is serialized in a long time, it should be the same amount of long time regardless of how many messages the sender is told to send to receiver. Am I missing something? Thank you! -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
On Mon, Mar 24, 2014 at 8:30 AM, Patrik Nordwall patrik.nordw...@gmail.comwrote: Hi Boris, Thanks for packing it in a reproducible sample. As Viktor points out the Java serializer is one of the prime suspect for bottlenecks in remote communication. Here you have found something that I think can be improved in Akka remoting. I can see that major time is consumed in EndpointWriter stashing and unstashing messages. That happens when the transport can't write. Perhaps some internal buffer is full. Please create a tickethttp://doc.akka.io/docs/akka/current/project/issue-tracking.htmland we will look into if we can improve the efficiency of this buffering mechanism. The TCP write buffers should be probably increased in this case because the EndpointWriter waits for Netty to pump out data in the background. This setting is at akka.remote.netty.tcp.send-buffer-size, by default it is 256000b, which is quite small. Try to increase it first and see what happens. Also the thread-pool settings for Netty in akka.remote.netty.tcp.server-socket-worker-pool and akka.remote.netty.tcp.client-socket-worker-pool allocate only two threads by default, increase it and see what happens. What you should do is to implement flow control in the application layer to avoid producing faster than what can be consumed. That is another important piece of course. -Endre and also, implement a good serializer for your messages. /Patrik On Mon, Mar 24, 2014 at 1:58 AM, √iktor Ҡlang viktor.kl...@gmail.comwrote: Hello Boris, Then you are likely to benchmark the slowness of Java Serialization. To maximize performance, one needs to choose an appropriate serializer. On Mon, Mar 24, 2014 at 1:51 AM, Boris Capitanu bor...@gmail.com wrote: Hello Viktor, I am using whatever default serializer Akka comes with. I have not configured anything special for that. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ * ——— **Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe http://typesafe.com/ - Reactive apps on the JVM Twitter: @patriknw -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Neither networked nor multithreaded applications have constant performance. Thread scheduling artifacts, cache effects, timing issues in handoffs, network congestion, retransmissions etc. Yes, that all makes sense... except that the network congestion and retransmission part should not have been an issue in my experiment. The runs were done on the same machine in different JVM using localhost, so there shouldn't be a need for TCP to retransmit anything... and according to the profiler the network was not anywhere close to being maxed out (looking at the socket read and write graphs). I'll test the settings that Endre pointed out and see what difference those settings make. Thank you all, again, for your help! -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
I have created the ticket that I still think is valid: https://www.assembla.com/spaces/akka/tickets/3960 /Patrik On Mon, Mar 24, 2014 at 3:17 PM, Akka Team akka.offic...@gmail.com wrote: Hi Boris, I'll test the settings that Endre pointed out and see what difference those settings make. Please report back what you have found. Also, I forgot to mention, but there is another setting that controls the thread-pool (dispatcher) for the remoting subsystem (this is independent of the netty pools). You should probably increase the size of that as well. The setting is under: akka.remote.default-remote-dispatcher, see http://doc.akka.io/docs/akka/2.3.0/scala/remoting.html#Remote_Configurationfor the complete configuration. Since you run everything on localhost fairness issues might dominate the case: some subsystems getting less CPU share since they have less threads to run on, and fairness is enforced by the OS per thread. I am interested in the results! -Endre Thank you all, again, for your help! -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe http://typesafe.com/ - Reactive apps on the JVM Twitter: @patriknw -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Endre,
My preliminary tests playing with the following settings show no
improvement (but in some cases even worse performance):
send-buffer-size = X
server-socket-worker-pool = {
pool-size-min = Y
pool-size-max = Y
}
client-socket-worker-pool = {
pool-size-min = Z
pool-size-max = Z
}
I tried different values for X, Y, Z, such as X = 256000b, X=512000b,
X=1024000b... and Y, Z = 4 and 8
I tried the settings separately, and together... none of the combinations I
tried improved the performance of what I've been seeing before.
It looked that whenever I was increasing the value for X, the performance
was actually getting worse. While the changes to Y and Z in isolation made
no observable difference in the timings recorded.
there is another setting that controls the thread-pool (dispatcher) for the
remoting subsystem (this is independent of the netty pools). You should
probably increase the size of that as well. The setting is under:
akka.remote.default-remote-dispatcher, see
http://doc.akka.io/docs/akka/2.3.0/scala/remoting.html#Remote_Configurationfor
the complete configuration.
I will try that and report back.
Since you run everything on localhost fairness issues might dominate the
case: some subsystems getting less CPU share since they have less threads
to run on, and fairness is enforced by the OS per thread.
Well... again, the sender and receiver were ran in two separate JVM
processes, so the subsystems in the two different processes shouldn't be
competing with each other based on the configuration limits imposed by the
akka configuration used. The other problem might be competition for CPU
time on the machine itself... but if you look at the CPU utilization graphs
in the profiler, the CPU is also not nearly used at it's max capacity.
I'll try the dispatcher setting and report back...
Oh, what I forgot to mention is that none of the hypothesis we explored so
far seem to explain (in my mind at least) why the later sets of 20K
messages are being received progressively faster (more sample run outputs
to see that are available in the README on GitHub). In a standard
producer-consumer scenario where the producer is the sender actor (which
produced 1.6M messages) and the consumer is the akka remoting subsystem
(which consumes data from that queue to send out over the network), I would
expect the consumer to behave the same way while draining messages from the
queue.
Regarding flow control --- I don't want to get to that yet... I'm not even
sure how I would know how to detect/trigger flow control. In
(oversimplified) traditional flow control the receiver detects that it's
being overwhelmed and informs the sender to rate-limit its output until the
receiver is capable of processing more messages...etc. In my case the
receiver actor (the application layer) would gladly consume as much data as
it gets it's not being overwhelmed to the point where it should/could
be sending a message to the sender to tell it to slow down.
Here's the fun part... I noticed that if I add a Thread.sleep(5000) in
the for loop on the sender side after each 100K messages sent (commented
out now in the GitHub code) then sending even 1M messages is done at the
same timings per 20K as I see when I send just 100K messages (without the
thread.sleep).
-Boris
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups Akka
User List group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Thank you, Patrik. I think you hit the nail in the head with that ticket. I wanted to write earlier that this felt like behavior you see when you have two nested for loops: for i = N to 1 for j = 1 to N do_something The stash-unstash cycle you pointed out effectively behaves like the above example... and the reason later messages get sent out faster is because there is progressively less stashing to do. Is this a hard-to-implement fix? -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Boris,
Thank you for trying out! Patrik I think figured out the reason (described
in the ticket). This is very much related to backpressure since the
messages get piling up in the EndpointWriter which continuosly stashes and
unstashes an ever growing amount of messages (gets to O(n^2) eventually).
This behavior will not happen if messages are backpressured (that keeps
unstash work constant).
Anyway, one thing to try is to set akka.remote.backoff-interval to a
larger value while setting the send-buffer-size to 1024000b. I would try
with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a
good way to test our hypothesis.
-Endre
On Mon, Mar 24, 2014 at 4:02 PM, Boris Capitanu bor...@gmail.com wrote:
Hi Endre,
My preliminary tests playing with the following settings show no
improvement (but in some cases even worse performance):
send-buffer-size = X
server-socket-worker-pool = {
pool-size-min = Y
pool-size-max = Y
}
client-socket-worker-pool = {
pool-size-min = Z
pool-size-max = Z
}
I tried different values for X, Y, Z, such as X = 256000b, X=512000b,
X=1024000b... and Y, Z = 4 and 8
I tried the settings separately, and together... none of the combinations
I tried improved the performance of what I've been seeing before.
It looked that whenever I was increasing the value for X, the performance
was actually getting worse. While the changes to Y and Z in isolation made
no observable difference in the timings recorded.
there is another setting that controls the thread-pool (dispatcher) for
the remoting subsystem (this is independent of the netty pools). You should
probably increase the size of that as well. The setting is under:
akka.remote.default-remote-dispatcher, see
http://doc.akka.io/docs/akka/2.3.0/scala/remoting.html#
Remote_Configuration for the complete configuration.
I will try that and report back.
Since you run everything on localhost fairness issues might dominate the
case: some subsystems getting less CPU share since they have less threads
to run on, and fairness is enforced by the OS per thread.
Well... again, the sender and receiver were ran in two separate JVM
processes, so the subsystems in the two different processes shouldn't be
competing with each other based on the configuration limits imposed by the
akka configuration used. The other problem might be competition for CPU
time on the machine itself... but if you look at the CPU utilization graphs
in the profiler, the CPU is also not nearly used at it's max capacity.
I'll try the dispatcher setting and report back...
Oh, what I forgot to mention is that none of the hypothesis we explored so
far seem to explain (in my mind at least) why the later sets of 20K
messages are being received progressively faster (more sample run outputs
to see that are available in the README on GitHub). In a standard
producer-consumer scenario where the producer is the sender actor (which
produced 1.6M messages) and the consumer is the akka remoting subsystem
(which consumes data from that queue to send out over the network), I would
expect the consumer to behave the same way while draining messages from the
queue.
Regarding flow control --- I don't want to get to that yet... I'm not even
sure how I would know how to detect/trigger flow control. In
(oversimplified) traditional flow control the receiver detects that it's
being overwhelmed and informs the sender to rate-limit its output until the
receiver is capable of processing more messages...etc. In my case the
receiver actor (the application layer) would gladly consume as much data as
it gets it's not being overwhelmed to the point where it should/could
be sending a message to the sender to tell it to slow down.
Here's the fun part... I noticed that if I add a Thread.sleep(5000) in
the for loop on the sender side after each 100K messages sent (commented
out now in the GitHub code) then sending even 1M messages is done at the
same timings per 20K as I see when I send just 100K messages (without the
thread.sleep).
-Boris
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
Akka User List group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Boris, On Mon, Mar 24, 2014 at 4:19 PM, Boris Capitanu bor...@gmail.com wrote: Thank you, Patrik. I think you hit the nail in the head with that ticket. I wanted to write earlier that this felt like behavior you see when you have two nested for loops: for i = N to 1 for j = 1 to N do_something The stash-unstash cycle you pointed out effectively behaves like the above example... and the reason later messages get sent out faster is because there is progressively less stashing to do. Yes, I think this is the reason. Setting the backoff will reduce this somewhat, but it is not a fix -- just a way to quickly check what is happening. Is this a hard-to-implement fix? Not any more. It is hard to fix in 2.2.x because we relied on the mailbox's property to keep messages during restart of an actor to not drop messages betwen reconnect attempts. Since now Gate is the only reconnect behavior and that drops, we no longer need Stash. -Endre -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Oops... I meant the second for was supposed to be for j = 1 to i (not to N) :-) -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Anyway, one thing to try is to set akka.remote.backoff-interval to a larger value while setting the send-buffer-size to 1024000b. I would try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is a good way to test our hypothesis. I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and I do see the timings becoming more consistent (albeit worse). The standard deviation of the timings observed is much lower. Well - I think we narrowed down the issue. I'll wait for a fix... I'll be glad to test any nightly builds that include a fix if it would be helpful. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Patrik,
After further testing, I'm running into an issue I can't explain and I'm
hoping you could shed some light on it...
The simplest form that showcases the problem is the following setup:
Two actors... a Sender and a Receiver, using RemoteActorRefProvider. The
receiver is started on a known port and waits to receive case class
Message(id: Int) messages.
When the sender starts (as a different JVM process), it looks up the
ActorRef of the receiver, and then does: for i=1 to N { receiver !
Message(i) }
(the code for this is here:
https://github.com/borice/akka-remote-actor-flood-test)
What I'm seeing is that if I look at the amount of time it takes to receive
20K messages, that time depends on the value of N (which it shouldn't).
For example, when N=100,000, it takes about 500ms - 600ms to receive 20K
messages. When N=800,00, it takes 6-7 seconds (10x increase) per 20K
messages
The other weird thing is that the later batches of 20K messages are
received progressively faster.
If you look at the GitHub link above, the README.md file contains example
log output after running the application for different values of N.
I've ran both sender and receiver through a profiler (YourKit profiler) and
here's what I'm seeing:
For N = 100,000
https://lh5.googleusercontent.com/-StPZXZuvXf4/Uy9hGzSyIbI/J-c/NIpqSdavFTs/s3200/profile_100k.png
For N = 800,000 (stopped capturing profiler data after receiver received
300,000 messages)
https://lh5.googleusercontent.com/-u2dHrkV3YkM/Uy9d1zeaDKI/J-Q/16tlNRzagOk/s3200/profile_800k.png
From the socket send/receive graphs it looks like the sender isn't sending
the data out as fast as the network can handle... (in other words, I'm
inclined to think that something in the sending side is not taking
advantage of all the network bandwidth available. The CPU usage was not
being maxed out either... the garbage collector wasn't working overly
hard... so I'm not sure I understand why using a different value for N (the
number of messages sent) matters to the amount of time each block of
messages is received.
Does this make sense to you?
One other usage for Akka I was planning is to retrieve streaming data from
a Microsoft Kinect from a service sending it via UDP, then do some
calculations on it, and send it back out to someone else via UDP. So this
would be a live continuous stream of data coming at (for now) 30
readings/second... but soon perhaps at 100 readings/second.
Thanks very much for your help!!!
Best,
Boris
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups Akka
User List group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Boris,
what serializer are you using for your messages?
On Sun, Mar 23, 2014 at 11:39 PM, Boris Capitanu bor...@gmail.com wrote:
Hi Patrik,
After further testing, I'm running into an issue I can't explain and I'm
hoping you could shed some light on it...
The simplest form that showcases the problem is the following setup:
Two actors... a Sender and a Receiver, using RemoteActorRefProvider. The
receiver is started on a known port and waits to receive case class
Message(id: Int) messages.
When the sender starts (as a different JVM process), it looks up the
ActorRef of the receiver, and then does: for i=1 to N { receiver !
Message(i) }
(the code for this is here:
https://github.com/borice/akka-remote-actor-flood-test)
What I'm seeing is that if I look at the amount of time it takes to
receive 20K messages, that time depends on the value of N (which it
shouldn't).
For example, when N=100,000, it takes about 500ms - 600ms to receive 20K
messages. When N=800,00, it takes 6-7 seconds (10x increase) per 20K
messages
The other weird thing is that the later batches of 20K messages are
received progressively faster.
If you look at the GitHub link above, the README.md file contains example
log output after running the application for different values of N.
I've ran both sender and receiver through a profiler (YourKit profiler)
and here's what I'm seeing:
For N = 100,000
https://lh5.googleusercontent.com/-StPZXZuvXf4/Uy9hGzSyIbI/J-c/NIpqSdavFTs/s3200/profile_100k.png
For N = 800,000 (stopped capturing profiler data after receiver received
300,000 messages)
https://lh5.googleusercontent.com/-u2dHrkV3YkM/Uy9d1zeaDKI/J-Q/16tlNRzagOk/s3200/profile_800k.png
From the socket send/receive graphs it looks like the sender isn't sending
the data out as fast as the network can handle... (in other words, I'm
inclined to think that something in the sending side is not taking
advantage of all the network bandwidth available. The CPU usage was not
being maxed out either... the garbage collector wasn't working overly
hard... so I'm not sure I understand why using a different value for N (the
number of messages sent) matters to the amount of time each block of
messages is received.
Does this make sense to you?
One other usage for Akka I was planning is to retrieve streaming data from
a Microsoft Kinect from a service sending it via UDP, then do some
calculations on it, and send it back out to someone else via UDP. So this
would be a live continuous stream of data coming at (for now) 30
readings/second... but soon perhaps at 100 readings/second.
Thanks very much for your help!!!
Best,
Boris
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
Akka User List group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Cheers,
√
*———**Viktor Klang*
*Chief Architect - **Typesafe http://www.typesafe.com/*
Twitter: @viktorklang
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups Akka
User List group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hello Viktor, I am using whatever default serializer Akka comes with. I have not configured anything special for that. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hello Boris, Then you are likely to benchmark the slowness of Java Serialization. To maximize performance, one needs to choose an appropriate serializer. On Mon, Mar 24, 2014 at 1:51 AM, Boris Capitanu bor...@gmail.com wrote: Hello Viktor, I am using whatever default serializer Akka comes with. I have not configured anything special for that. -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, √ *———**Viktor Klang* *Chief Architect - **Typesafe http://www.typesafe.com/* Twitter: @viktorklang -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[akka-user] testing akka cluster on Blue Waters @ NCSA
Hello,
Apologies in advance for the long post... to frame my problem and questions
I have to give a lot of contextual info.
I am running a set of experiments to understand the feasibility and
performance characteristics of running Akka on the Blue
Watershttps://bluewaters.ncsa.illinois.edu/(batch) supercomputer at
NCSA http://www.ncsa.illinois.edu/.
The general theme for the experiments is to create a map-reduce like system
comprised of a producer, one (or more) coordinator(s), and a number of
workers. (the reduce step could be optional)
The producer is responsible for generating the work which is then sent to
the coordinator(s) which in turn assign the work to an idle worker.
At first I am most concerned with the performance of the solution (rather
than fault-tolerance). The only fault-tolerance aspect I'd like to address
at first consists of exceptions thrown as part of executing the work.
When that happens, the executor should be restarted and the coordinator
informed of the failed job which would get logged and perhaps retried later
(up to n times) depending on the exception. Eventually the final solution
would include a checkpointing mechanism that would allow the PBS job to be
resumed from the last checkpoint in case the it is killed or a malfunction
occurs with BlueWaters.
Besides learning about the different options/architectures that are
appropriate for an environment such as Blue Waters, the problem I
ultimately want to solve is to process approx. 3.2 million gzipped text
files (about 3.2TB of gzipped data). The processing could be any number
of things - feature extraction, OCR error correction, or other
text-processing routines as needed.
About my experience: I am no Scala and Akka expert but I do have a decent
amount of experience with both. I've taken Martin's functional programming
in scala course on Coursera, as well as the reactive programming course by
Martin, Erik, and Roland. I've also read through a number of scala and akka
books, tutorials, examples...etc. but, as with just about everything else
in life, there's always more to learn.
My first attempt was to adapt the Distributed Workers with Akka and
Scalahttp://www.typesafe.com/activator/template/akka-distributed-workers
example
just to see how that performs out of the box. In the process I've also
had to find a solution to be able to get the application deployed to the
Blue Waters Torque/PBS batch environment, which wasn't trivial... but I did
come up with 2 ways to do that that seem to work quite well (if anyone
wants to know how, I could write about that). I did manage to get the
example to work - tested it on a small subset of the data (with some
changes to WorkProducer) - but I felt that this solution was too overly
concerned with fault tolerance at the expense of (some) performance, so I
wanted to create another solution where those concerns would be reversed.
(it's still on my TODO list to get some performance measurements of the
original example so that I can compare numbers across the different
solutions)
The way I'm planning on evaluating the performance of each solution is by
having the work producer generate a known quantity of (empty) work as fast
as possible, have that work be coordinated by the master over a known
number of worker nodes, and have the executor on each worker do nothing
else but respond immediately with an (empty) result. From this setup I want
to record the timing of the following operations:
- overall wall clock time from start of work being produced to when the
last result was received by the worker actor from the executor (the master
and workers would already be up, registered, and ready before the producer
starts sending out work) (this time is not affected by other jobs running
on the bluewaters system since the work involves no I/O to disk - just
network - and the compute nodes where the worker actors are deployed are
exclusively dedicated to this task - are not running other user jobs)
- overall stats about the time elapsed between when a Work object is
sent to the master by the producer, and the moment the Master receives it
- overall stats about the time elapsed between Master sending work to
Worker, and Worker responding with WorkIsDone
In (pseudo-scala) code, the solution I have created behaves like this (the
relevant parts):
*Producer*
val numWorkToGenerate = N
for (i - 1 to numWorkToGenerate) {
timestamp = getCurrentTimestamp
work = new Work(i, timestamp)
master ! work
}
master ! NoMoreWork
*Master*
... keep track of workers and what they're working on, and keep a work
queue of work that still needs to be assigned ...
def receive = {
case work: Work =
record time delta based on current timestamp and timestamp packaged
in 'work'
find idle worker and, if found, update timstamp in 'work' with
current timestamp and send 'work' to worker
case
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Boris,
On Sat, Mar 22, 2014 at 6:45 AM, Boris Capitanu bor...@gmail.com wrote:
Hello,
Apologies in advance for the long post... to frame my problem and
questions I have to give a lot of contextual info.
I am running a set of experiments to understand the feasibility and
performance characteristics of running Akka on the Blue
Watershttps://bluewaters.ncsa.illinois.edu/(batch) supercomputer at
NCSA http://www.ncsa.illinois.edu/.
The general theme for the experiments is to create a map-reduce like
system comprised of a producer, one (or more) coordinator(s), and a number
of workers. (the reduce step could be optional)
The producer is responsible for generating the work which is then sent
to the coordinator(s) which in turn assign the work to an idle worker.
At first I am most concerned with the performance of the solution (rather
than fault-tolerance). The only fault-tolerance aspect I'd like to address
at first consists of exceptions thrown as part of executing the work.
When that happens, the executor should be restarted and the coordinator
informed of the failed job which would get logged and perhaps retried later
(up to n times) depending on the exception. Eventually the final solution
would include a checkpointing mechanism that would allow the PBS job to be
resumed from the last checkpoint in case the it is killed or a malfunction
occurs with BlueWaters.
Besides learning about the different options/architectures that are
appropriate for an environment such as Blue Waters, the problem I
ultimately want to solve is to process approx. 3.2 million gzipped text
files (about 3.2TB of gzipped data). The processing could be any number
of things - feature extraction, OCR error correction, or other
text-processing routines as needed.
About my experience: I am no Scala and Akka expert but I do have a decent
amount of experience with both. I've taken Martin's functional programming
in scala course on Coursera, as well as the reactive programming course by
Martin, Erik, and Roland. I've also read through a number of scala and akka
books, tutorials, examples...etc. but, as with just about everything else
in life, there's always more to learn.
My first attempt was to adapt the Distributed Workers with Akka and
Scalahttp://www.typesafe.com/activator/template/akka-distributed-workers
example
just to see how that performs out of the box. In the process I've also
had to find a solution to be able to get the application deployed to the
Blue Waters Torque/PBS batch environment, which wasn't trivial... but I did
come up with 2 ways to do that that seem to work quite well (if anyone
wants to know how, I could write about that). I did manage to get the
example to work - tested it on a small subset of the data (with some
changes to WorkProducer) - but I felt that this solution was too overly
concerned with fault tolerance at the expense of (some) performance, so I
wanted to create another solution where those concerns would be reversed.
(it's still on my TODO list to get some performance measurements of the
original example so that I can compare numbers across the different
solutions)
The way I'm planning on evaluating the performance of each solution is by
having the work producer generate a known quantity of (empty) work as fast
as possible, have that work be coordinated by the master over a known
number of worker nodes, and have the executor on each worker do nothing
else but respond immediately with an (empty) result.
Be aware of that the network roundtrips might dominate when you test with
empty work in a way that might not be relevant when the workers do real
work.
From this setup I want to record the timing of the following operations:
- overall wall clock time from start of work being produced to when
the last result was received by the worker actor from the executor (the
master and workers would already be up, registered, and ready before the
producer starts sending out work) (this time is not affected by other jobs
running on the bluewaters system since the work involves no I/O to disk -
just network - and the compute nodes where the worker actors are deployed
are exclusively dedicated to this task - are not running other user jobs)
- overall stats about the time elapsed between when a Work object is
sent to the master by the producer, and the moment the Master receives it
- overall stats about the time elapsed between Master sending work to
Worker, and Worker responding with WorkIsDone
In (pseudo-scala) code, the solution I have created behaves like this (the
relevant parts):
*Producer*
val numWorkToGenerate = N
for (i - 1 to numWorkToGenerate) {
timestamp = getCurrentTimestamp
work = new Work(i, timestamp)
master ! work
}
master ! NoMoreWork
*Master*
... keep track of workers and what they're working on, and keep a work
queue of work that still needs to be
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
Hi Patrik, Thanks for your prompt response! Be aware of that the network roundtrips might dominate when you test with empty work in a way that might not be relevant when the workers do real work. Right, I realize that, but I just wanted to get some baseline timing information for the absolute minimum amount of time needed to run X (empty) jobs. I guess I was just trying to measure the overhead of the system. I could simulate work taking some (fixed) time by inserting a (i know, bad!) Thread.sleep in the executor... I suppose that might make the system more fluent since the network will become less congested. I'll try that and see what effect it has... For those interested the actual code is here: https://github.com/borice/akka-grid-workers I noticed one error when use onSuccess callback. See here http://doc.akka.io/docs/akka/2.3.0/general/jmm.html#Actors_and_shared_mutable_state Thank you, fixed. Given the way messages were exchanged I didn't think that was a problem, but I was just being lazy... :) long garbage collection pauses can also trigger failure detection That makes sense... I wonder whether the akka cluster extension in this case could be playing a part in the issues I'm seeing. The heartbeats or gossip messages exchanged between the cluster nodes might add unneeded traffic to a potentially-already-saturated network. I think that the simple solution I presented above could be implemented without using the cluster extension, by just using remote actors. The Producer only needs to know about the Coordinator (and monitor its lifecycle), and similar for the Workers. I wonder what the practical limits are for the maximum size of an akka cluster, assuming there are no JVM-related limits, when the only limit becomes the communication protocol/medium... http://typesafe.com/blog/running-a-2400-akka-nodes-cluster-on-google-compute-enginehttp://www.google.com/url?q=http%3A%2F%2Ftypesafe.com%2Fblog%2Frunning-a-2400-akka-nodes-cluster-on-google-compute-enginesa=Dsntz=1usg=AFQjCNE7AOab4aK84Q3vwCpYoGm47VuzzQ Very cool! I'll try to use some of the same tuning adjustments to the configuration settings that were described in that article. They will likely alleviate some of the issues I'm seeing... I'll see if I'll be able to run some tests across a 2400 node akka cluster... if I do, I'll write about it here. Thanks again! Scala + Akka is an absolute pleasure to work with... -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] testing akka cluster on Blue Waters @ NCSA
22 mar 2014 kl. 16:49 skrev Boris Capitanu bor...@gmail.com: Hi Patrik, Thanks for your prompt response! Be aware of that the network roundtrips might dominate when you test with empty work in a way that might not be relevant when the workers do real work. Right, I realize that, but I just wanted to get some baseline timing information for the absolute minimum amount of time needed to run X (empty) jobs. I guess I was just trying to measure the overhead of the system. I could simulate work taking some (fixed) time by inserting a (i know, bad!) Thread.sleep in the executor... I suppose that might make the system more fluent since the network will become less congested. I'll try that and see what effect it has... For those interested the actual code is here: https://github.com/borice/akka-grid-workers I noticed one error when use onSuccess callback. See here http://doc.akka.io/docs/akka/2.3.0/general/jmm.html#Actors_and_shared_mutable_state Thank you, fixed. Given the way messages were exchanged I didn't think that was a problem, but I was just being lazy... :) long garbage collection pauses can also trigger failure detection That makes sense... I wonder whether the akka cluster extension in this case could be playing a part in the issues I'm seeing. The heartbeats or gossip messages exchanged between the cluster nodes might add unneeded traffic to a potentially-already-saturated network. I think that the simple solution I presented above could be implemented without using the cluster extension, by just using remote actors. The Producer only needs to know about the Coordinator (and monitor its lifecycle), and similar for the Workers. I wonder what the practical limits are for the maximum size of an akka cluster, assuming there are no JVM-related limits, when the only limit becomes the communication protocol/medium... http://typesafe.com/blog/running-a-2400-akka-nodes-cluster-on-google-compute-engine Very cool! I'll try to use some of the same tuning adjustments to the configuration settings that were described in that article. They will likely alleviate some of the issues I'm seeing... I'll see if I'll be able to run some tests across a 2400 node akka cluster... if I do, I'll write about it here. Thanks again! Scala + Akka is an absolute pleasure to work with... I'm glad to hear that. Looking forward to see your results, or questions. /Patrik -Boris -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
