I finally narrowed down the config item: akka.actor.remote.use-dispatcher
The default setting for akka 2.3.4 remote is
akka.actor.remote.use-dispatcher = "akka.remote.default-remote-dispatcher",
when I change it to akka.actor.remote.use-dispatcher = "", then the
performance is same or better with akka 2.2.3.
On Thursday, August 7, 2014 7:57:28 PM UTC+8, √ wrote:
>
> I hope you didn't try to run the config verbatim as it works differently
> between the versions, but what happened when you updated to use the 2.2.3
> values for the options that were still there in 2.3.4?
>
>
> On Thu, Aug 7, 2014 at 1:55 PM, Sean Zhong <cloc...@gmail.com
> <javascript:>> wrote:
>
>> I made a diff, and try to use old akka 2.2.3 config when running with
>> akka 2.3.4
>>
>> Here is the diff:
>>
>> --- akka.2.3.4.conf.json Thu Aug 7 19:33:37 2014
>>> +++ akka2.2.3.conf.json Thu Aug 7 19:32:35 2014
>>> @@ -13,13 +13,10 @@
>>> },
>>> "default-dispatcher": {
>>> "attempt-teamwork": "on",
>>> - "default-executor": {
>>> - "fallback": "fork-join-executor"
>>> - },
>>> - "executor": "default-executor",
>>> + "executor": "fork-join-executor",
>>> "fork-join-executor": {
>>> "parallelism-factor": 3,
>>> - "parallelism-max": 4,
>>> + "parallelism-max": 64,
>>> "parallelism-min": 8
>>> },
>>> "mailbox-capacity": -1,
>>> @@ -40,14 +37,14 @@
>>> "task-queue-size": -1,
>>> "task-queue-type": "linked"
>>> },
>>> - "throughput": 1024,
>>> + "throughput": 5,
>>> "throughput-deadline-time": "0ms",
>>> "type": "Dispatcher"
>>> },
>>> "default-mailbox": {
>>> "mailbox-capacity": 1000,
>>> "mailbox-push-timeout-time": "10s",
>>> - "mailbox-type":
>>> "akka.dispatch.SingleConsumerOnlyUnboundedMailbox",
>>> + "mailbox-type": "akka.dispatch.UnboundedMailbox",
>>> "stash-capacity": -1
>>> },
>>> "default-stash-dispatcher": {
>>> @@ -62,7 +59,6 @@
>>> "resizer": {
>>> "backoff-rate": 0.1,
>>> "backoff-threshold": 0.3,
>>> - "enabled": "off",
>>> "lower-bound": 1,
>>> "messages-per-resize": 10,
>>> "pressure-threshold": 1,
>>> @@ -110,29 +106,12 @@
>>> },
>>> "provider": "akka.remote.RemoteActorRefProvider",
>>> "reaper-interval": "5s",
>>> - "router": {
>>> - "type-mapping": {
>>> - "balancing-pool": "akka.routing.BalancingPool",
>>> - "broadcast-group": "akka.routing.BroadcastGroup",
>>> - "broadcast-pool": "akka.routing.BroadcastPool",
>>> - "consistent-hashing-group":
>>> "akka.routing.ConsistentHashingGroup",
>>> - "consistent-hashing-pool":
>>> "akka.routing.ConsistentHashingPool",
>>> - "from-code": "akka.routing.NoRouter",
>>> - "random-group": "akka.routing.RandomGroup",
>>> - "random-pool": "akka.routing.RandomPool",
>>> - "round-robin-group": "akka.routing.RoundRobinGroup",
>>> - "round-robin-pool": "akka.routing.RoundRobinPool",
>>> - "scatter-gather-group":
>>> "akka.routing.ScatterGatherFirstCompletedGroup",
>>> - "scatter-gather-pool":
>>> "akka.routing.ScatterGatherFirstCompletedPool",
>>> - "smallest-mailbox-pool": "akka.routing.SmallestMailboxPool"
>>> - }
>>> - },
>>> "serialization-bindings": {
>>> "[B": "bytes",
>>> - "akka.actor.ActorSelectionMessage": "akka-containers",
>>> + "akka.actor.SelectionPath": "akka-containers",
>>> "akka.remote.DaemonMsgCreate": "daemon-create",
>>> - "com.google.protobuf.GeneratedMessage": "proto",
>>> - "java.io.Serializable": "java",
>>> + "com.google.protobuf_spark.GeneratedMessage": "proto",
>>> + "java.io.Serializable": "java"
>>> },
>>> "serialize-creators": "off",
>>> "serialize-messages": "off",
>>> @@ -149,13 +128,9 @@
>>> "unstarted-push-timeout": "10s"
>>> },
>>> "daemonic": "off",
>>> - "event-handler-startup-timeout": "5s",
>>> - "event-handlers": [
>>> - "akka.event.Logging$DefaultLogger"
>>> - ],
>>> - "extensions": [
>>> - "com.romix.akka.serialization.kryo.KryoSerializationExtension$"
>>> - ],
>>> + "event-handler-startup-timeout": "-1s",
>>> + "event-handlers": [],
>>> + "extensions": [],
>>> "home": "",
>>> "io": {
>>> "default-backlog": 1000,
>>> @@ -228,37 +203,18 @@
>>> "gremlin": "akka.remote.transport.FailureInjectorProvider",
>>> "trttl": "akka.remote.transport.ThrottlerProvider"
>>> },
>>> - "backoff-interval": "5 ms",
>>> - "backoff-remote-dispatcher": {
>>> - "executor": "fork-join-executor",
>>> - "fork-join-executor": {
>>> - "parallelism-max": 2,
>>> - "parallelism-min": 2
>>> - },
>>> - "type": "Dispatcher"
>>> - },
>>> + "backoff-interval": "0.01 s",
>>> "command-ack-timeout": "30 s",
>>> - "default-remote-dispatcher": {
>>> - "executor": "fork-join-executor",
>>> - "fork-join-executor": {
>>> - "parallelism-max": 2,
>>> - "parallelism-min": 2
>>> - },
>>> - "type": "Dispatcher"
>>> - },
>>> "enabled-transports": [
>>> "akka.remote.netty.tcp"
>>> ],
>>> "flush-wait-on-shutdown": "2 s",
>>> - "gremlin": {
>>> - "debug": "off"
>>> - },
>>> - "initial-system-message-delivery-timeout": "3 m",
>>> - "log-buffer-size-exceeding": 50000,
>>> + "gate-invalid-addresses-for": "60 s",
>>> "log-frame-size-exceeding": "off",
>>> "log-received-messages": "off",
>>> "log-remote-lifecycle-events": "on",
>>> "log-sent-messages": "off",
>>> + "maximum-retries-in-window": 3,
>>> "netty": {
>>> "ssl": {
>>> "applied-adapters": [],
>>> @@ -360,26 +316,29 @@
>>> "write-buffer-low-water-mark": "0b"
>>> }
>>> },
>>> - "prune-quarantine-marker-after": "5 d",
>>> + "quarantine-systems-for": "60s",
>>> "require-cookie": "off",
>>> - "resend-interval": "2 s",
>>> - "retry-gate-closed-for": "5 s",
>>> + "resend-interval": "1 s",
>>> + "retry-gate-closed-for": "0 s",
>>> + "retry-window": "60 s",
>>> "secure-cookie": "",
>>> "shutdown-timeout": "10 s",
>>> "startup-timeout": "10 s",
>>> - "system-message-ack-piggyback-timeout": "0.3 s",
>>> + "system-message-ack-piggyback-timeout": "1 s",
>>> "system-message-buffer-size": 1000,
>>> "transport-failure-detector": {
>>> - "acceptable-heartbeat-pause": "20 s",
>>> - "heartbeat-interval": "4 s",
>>> - "implementation-class": "akka.remote.DeadlineFailureDetector"
>>> + "acceptable-heartbeat-pause": "3 s",
>>> + "heartbeat-interval": "1 s",
>>> + "implementation-class": "akka.remote.PhiAccrualFailureDetector",
>>> + "max-sample-size": 100,
>>> + "min-std-deviation": "100 ms",
>>> + "threshold": 7
>>> },
>>> - "trusted-selection-paths": [],
>>> "untrusted-mode": "off",
>>> - "use-dispatcher": "akka.remote.default-remote-dispatcher",
>>> + "use-dispatcher": "",
>>> "use-passive-connections": "on",
>>> "watch-failure-detector": {
>>> - "acceptable-heartbeat-pause": "10 s",
>>> + "acceptable-heartbeat-pause": "4 s",
>>> "expected-response-after": "3 s",
>>> "heartbeat-interval": "1 s",
>>> "implementation-class": "akka.remote.PhiAccrualFailureDetector",
>>> @@ -396,6 +355,6 @@
>>> "ticks-per-wheel": 512
>>> },
>>> "stdout-loglevel": "WARNING",
>>> - "version": "2.3.4"
>>> + "version": "2.2.3"
>>> }
>>> }
>>
>>
>>
>>
>> On Thursday, August 7, 2014 7:36:27 PM UTC+8, Sean Zhong wrote:
>>>
>>>
>>>
>>>
>>>
>>> <https://lh5.googleusercontent.com/-HoimNVHLEVs/U-NbcKNJOYI/AAAAAAAAD7Q/kESkTGWcAdQ/s1600/001.png>
>>>
>>> Cluster: 4 machines, each machine has 1 source actor and 1 target
>>> actor.. all actor started remotely by a master.
>>> test scenario: Each source actor will randomly deliver a 100 bytes
>>> messge at a time to any target actor.
>>>
>>> Test result：
>>>
>>> akka 2.2.3: 500K 100bytes message /second
>>> akka 2.3.4: 320K 100bytes message / second
>>>
>>> the network bandwidth occupation reflects the message throughput.
>>> network bandwidth is 3.2x message throughput. (as the actorPath overhead is
>>> 221 bytes, aka 2.2 times the message size in my test)
>>>
>>> Network bandwidth usage ratio: akka2.2.3/akka 2.3.4 ~= 1.5
>>>
>>> But the CPU ratio akka2.2.3/akka 2.3.4 = 40/15 = 2.6.
>>>
>>> akka 2.2.3 use much more CPU for same throughput, does this mean akka
>>> 2.3.4 is more efficient? But why the message throughput drops on akka 2.3.4.
>>>
>>> I dumped the effective akka conf for 2.2.3 and 2.3.4 in the attachment.
>>>
>>>
>>> On Thursday, August 7, 2014 6:05:54 PM UTC+8, Sean Zhong wrote:
>>>>
>>>> Can it be the case that you have a lot of system message traffic
>>>>> between your systems? Do you have lots of remote deployed actors maybe?
>>>>
>>>>
>>>> All actors(4 source, 4 target) are created using remote actors.
>>>>
>>>>
>>>> On Thursday, August 7, 2014 5:45:34 PM UTC+8, drewhk wrote:
>>>>>
>>>>> Hi Sean,
>>>>>
>>>>> This is interesting, we actually measured increase with 2.3.4, in fact
>>>>> the relevant changes in that version were directly targeted to increasing
>>>>> performance somewhat. One important difference is that 2.3.4 prioritizes
>>>>> internal Akka messages over user messages to improve stability. Can it be
>>>>> the case that you have a lot of system message traffic between your
>>>>> systems? Do you have lots of remote deployed actors maybe?
>>>>>
>>>>> -Endre
>>>>>
>>>>>
>>>>> On Thu, Aug 7, 2014 at 10:53 AM, Sean Zhong <cloc...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> When I upgrade from akka 2.2.3 to akka 2.3.4, I found the message
>>>>>> throughput drops about 30%.
>>>>>>
>>>>>> My benchmark looks like this:
>>>>>> 4 machines, each machine has 1 source actor and 1 target actor. Each
>>>>>> source actor will randomly deliver a 100 bytes messge at a time to any
>>>>>> target actor.
>>>>>>
>>>>>> I use default configuration. Are there default configuration changes
>>>>>> in akka reference.conf which result in this behavior?
>>>>>>
>>>>>> --
>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/
>>>>>> current/additional/faq.html
>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/
>>>>>> group/akka-user
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Akka User List" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+...@googlegroups.com <javascript:>.
>> To post to this group, send email to akka...@googlegroups.com
>> <javascript:>.
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Cheers,
> √
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
