Sean,
Yes the separate dispatcher might be the cause. This default was added to
protect the remoting subsystem from load in userspace (learning from some
past problems). Feel free to reconfigure it to anything that works for you
-- the default might be conservative indeed.
-Endre
On Thu, Aug 7, 2014 at 2:47 PM, Sean Zhong <clock...@gmail.com> wrote:
> The default-remote-dispatcher config is:
> ### Default dispatcher for the remoting subsystem
>
> default-remote-dispatcher {
> type = Dispatcher
> executor = "fork-join-executor"
> fork-join-executor {
> # Min number of threads to cap factor-based parallelism number to
> parallelism-min = 2
> parallelism-max = 2
> }
> }
>
> Is it possible that the value setting here are too Conservative which
> impact performance. I will tune this and see what happens...
>
>
>
> On Thursday, August 7, 2014 8:35:50 PM UTC+8, Sean Zhong wrote:
>>
>> I finally narrowed down the config item: akka.actor.remote.use-dispatcher
>>
>> The default setting for akka 2.3.4 remote is akka.actor.remote.use-dispatcher
>> = "akka.remote.default-remote-dispatcher", when I change it
>> to akka.actor.remote.use-dispatcher = "", then the performance is same
>> or better with akka 2.2.3.
>>
>>
>>
>>
>> On Thursday, August 7, 2014 7:57:28 PM UTC+8, √ wrote:
>>>
>>> I hope you didn't try to run the config verbatim as it works differently
>>> between the versions, but what happened when you updated to use the 2.2.3
>>> values for the options that were still there in 2.3.4?
>>>
>>>
>>> On Thu, Aug 7, 2014 at 1:55 PM, Sean Zhong <cloc...@gmail.com> wrote:
>>>
>>>> I made a diff, and try to use old akka 2.2.3 config when running with
>>>> akka 2.3.4
>>>>
>>>> Here is the diff:
>>>>
>>>> --- akka.2.3.4.conf.json Thu Aug 7 19:33:37 2014
>>>>> +++ akka2.2.3.conf.json Thu Aug 7 19:32:35 2014
>>>>> @@ -13,13 +13,10 @@
>>>>> },
>>>>> "default-dispatcher": {
>>>>> "attempt-teamwork": "on",
>>>>> - "default-executor": {
>>>>> - "fallback": "fork-join-executor"
>>>>> - },
>>>>> - "executor": "default-executor",
>>>>> + "executor": "fork-join-executor",
>>>>> "fork-join-executor": {
>>>>> "parallelism-factor": 3,
>>>>> - "parallelism-max": 4,
>>>>> + "parallelism-max": 64,
>>>>> "parallelism-min": 8
>>>>> },
>>>>> "mailbox-capacity": -1,
>>>>> @@ -40,14 +37,14 @@
>>>>> "task-queue-size": -1,
>>>>> "task-queue-type": "linked"
>>>>> },
>>>>> - "throughput": 1024,
>>>>> + "throughput": 5,
>>>>> "throughput-deadline-time": "0ms",
>>>>> "type": "Dispatcher"
>>>>> },
>>>>> "default-mailbox": {
>>>>> "mailbox-capacity": 1000,
>>>>> "mailbox-push-timeout-time": "10s",
>>>>> - "mailbox-type": "akka.dispatch.SingleConsumerOnlyUnboundedMai
>>>>> lbox",
>>>>> + "mailbox-type": "akka.dispatch.UnboundedMailbox",
>>>>> "stash-capacity": -1
>>>>> },
>>>>> "default-stash-dispatcher": {
>>>>> @@ -62,7 +59,6 @@
>>>>> "resizer": {
>>>>> "backoff-rate": 0.1,
>>>>> "backoff-threshold": 0.3,
>>>>> - "enabled": "off",
>>>>> "lower-bound": 1,
>>>>> "messages-per-resize": 10,
>>>>> "pressure-threshold": 1,
>>>>> @@ -110,29 +106,12 @@
>>>>> },
>>>>> "provider": "akka.remote.RemoteActorRefProvider",
>>>>> "reaper-interval": "5s",
>>>>> - "router": {
>>>>> - "type-mapping": {
>>>>> - "balancing-pool": "akka.routing.BalancingPool",
>>>>> - "broadcast-group": "akka.routing.BroadcastGroup",
>>>>> - "broadcast-pool": "akka.routing.BroadcastPool",
>>>>> - "consistent-hashing-group": "akka.routing.
>>>>> ConsistentHashingGroup",
>>>>> - "consistent-hashing-pool": "akka.routing.
>>>>> ConsistentHashingPool",
>>>>> - "from-code": "akka.routing.NoRouter",
>>>>> - "random-group": "akka.routing.RandomGroup",
>>>>> - "random-pool": "akka.routing.RandomPool",
>>>>> - "round-robin-group": "akka.routing.RoundRobinGroup",
>>>>> - "round-robin-pool": "akka.routing.RoundRobinPool",
>>>>> - "scatter-gather-group": "akka.routing.
>>>>> ScatterGatherFirstCompletedGroup",
>>>>> - "scatter-gather-pool": "akka.routing.
>>>>> ScatterGatherFirstCompletedPool",
>>>>> - "smallest-mailbox-pool": "akka.routing.SmallestMailboxPool"
>>>>> - }
>>>>> - },
>>>>> "serialization-bindings": {
>>>>> "[B": "bytes",
>>>>> - "akka.actor.ActorSelectionMessage": "akka-containers",
>>>>> + "akka.actor.SelectionPath": "akka-containers",
>>>>> "akka.remote.DaemonMsgCreate": "daemon-create",
>>>>> - "com.google.protobuf.GeneratedMessage": "proto",
>>>>> - "java.io.Serializable": "java",
>>>>> + "com.google.protobuf_spark.GeneratedMessage": "proto",
>>>>> + "java.io.Serializable": "java"
>>>>> },
>>>>> "serialize-creators": "off",
>>>>> "serialize-messages": "off",
>>>>> @@ -149,13 +128,9 @@
>>>>> "unstarted-push-timeout": "10s"
>>>>> },
>>>>> "daemonic": "off",
>>>>> - "event-handler-startup-timeout": "5s",
>>>>> - "event-handlers": [
>>>>> - "akka.event.Logging$DefaultLogger"
>>>>> - ],
>>>>> - "extensions": [
>>>>> - "com.romix.akka.serialization.kryo.KryoSerializationExtension$"
>>>>> - ],
>>>>> + "event-handler-startup-timeout": "-1s",
>>>>> + "event-handlers": [],
>>>>> + "extensions": [],
>>>>> "home": "",
>>>>> "io": {
>>>>> "default-backlog": 1000,
>>>>> @@ -228,37 +203,18 @@
>>>>> "gremlin": "akka.remote.transport.FailureInjectorProvider",
>>>>> "trttl": "akka.remote.transport.ThrottlerProvider"
>>>>> },
>>>>> - "backoff-interval": "5 ms",
>>>>> - "backoff-remote-dispatcher": {
>>>>> - "executor": "fork-join-executor",
>>>>> - "fork-join-executor": {
>>>>> - "parallelism-max": 2,
>>>>> - "parallelism-min": 2
>>>>> - },
>>>>> - "type": "Dispatcher"
>>>>> - },
>>>>> + "backoff-interval": "0.01 s",
>>>>> "command-ack-timeout": "30 s",
>>>>> - "default-remote-dispatcher": {
>>>>> - "executor": "fork-join-executor",
>>>>> - "fork-join-executor": {
>>>>> - "parallelism-max": 2,
>>>>> - "parallelism-min": 2
>>>>> - },
>>>>> - "type": "Dispatcher"
>>>>> - },
>>>>> "enabled-transports": [
>>>>> "akka.remote.netty.tcp"
>>>>> ],
>>>>> "flush-wait-on-shutdown": "2 s",
>>>>> - "gremlin": {
>>>>> - "debug": "off"
>>>>> - },
>>>>> - "initial-system-message-delivery-timeout": "3 m",
>>>>> - "log-buffer-size-exceeding": 50000,
>>>>> + "gate-invalid-addresses-for": "60 s",
>>>>> "log-frame-size-exceeding": "off",
>>>>> "log-received-messages": "off",
>>>>> "log-remote-lifecycle-events": "on",
>>>>> "log-sent-messages": "off",
>>>>> + "maximum-retries-in-window": 3,
>>>>> "netty": {
>>>>> "ssl": {
>>>>> "applied-adapters": [],
>>>>> @@ -360,26 +316,29 @@
>>>>> "write-buffer-low-water-mark": "0b"
>>>>> }
>>>>> },
>>>>> - "prune-quarantine-marker-after": "5 d",
>>>>> + "quarantine-systems-for": "60s",
>>>>> "require-cookie": "off",
>>>>> - "resend-interval": "2 s",
>>>>> - "retry-gate-closed-for": "5 s",
>>>>> + "resend-interval": "1 s",
>>>>> + "retry-gate-closed-for": "0 s",
>>>>> + "retry-window": "60 s",
>>>>> "secure-cookie": "",
>>>>> "shutdown-timeout": "10 s",
>>>>> "startup-timeout": "10 s",
>>>>> - "system-message-ack-piggyback-timeout": "0.3 s",
>>>>> + "system-message-ack-piggyback-timeout": "1 s",
>>>>> "system-message-buffer-size": 1000,
>>>>> "transport-failure-detector": {
>>>>> - "acceptable-heartbeat-pause": "20 s",
>>>>> - "heartbeat-interval": "4 s",
>>>>> - "implementation-class": "akka.remote.DeadlineFailureDetector"
>>>>> + "acceptable-heartbeat-pause": "3 s",
>>>>> + "heartbeat-interval": "1 s",
>>>>> + "implementation-class": "akka.remote.
>>>>> PhiAccrualFailureDetector",
>>>>> + "max-sample-size": 100,
>>>>> + "min-std-deviation": "100 ms",
>>>>> + "threshold": 7
>>>>> },
>>>>> - "trusted-selection-paths": [],
>>>>> "untrusted-mode": "off",
>>>>> - "use-dispatcher": "akka.remote.default-remote-dispatcher",
>>>>> + "use-dispatcher": "",
>>>>> "use-passive-connections": "on",
>>>>> "watch-failure-detector": {
>>>>> - "acceptable-heartbeat-pause": "10 s",
>>>>> + "acceptable-heartbeat-pause": "4 s",
>>>>> "expected-response-after": "3 s",
>>>>> "heartbeat-interval": "1 s",
>>>>> "implementation-class": "akka.remote.
>>>>> PhiAccrualFailureDetector",
>>>>> @@ -396,6 +355,6 @@
>>>>> "ticks-per-wheel": 512
>>>>> },
>>>>> "stdout-loglevel": "WARNING",
>>>>> - "version": "2.3.4"
>>>>> + "version": "2.2.3"
>>>>> }
>>>>> }
>>>>
>>>>
>>>>
>>>>
>>>> On Thursday, August 7, 2014 7:36:27 PM UTC+8, Sean Zhong wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> <https://lh5.googleusercontent.com/-HoimNVHLEVs/U-NbcKNJOYI/AAAAAAAAD7Q/kESkTGWcAdQ/s1600/001.png>
>>>>>
>>>>> Cluster: 4 machines, each machine has 1 source actor and 1 target
>>>>> actor.. all actor started remotely by a master.
>>>>> test scenario: Each source actor will randomly deliver a 100 bytes
>>>>> messge at a time to any target actor.
>>>>>
>>>>> Test result：
>>>>>
>>>>> akka 2.2.3: 500K 100bytes message /second
>>>>> akka 2.3.4: 320K 100bytes message / second
>>>>>
>>>>> the network bandwidth occupation reflects the message throughput.
>>>>> network bandwidth is 3.2x message throughput. (as the actorPath overhead
>>>>> is
>>>>> 221 bytes, aka 2.2 times the message size in my test)
>>>>>
>>>>> Network bandwidth usage ratio: akka2.2.3/akka 2.3.4 ~= 1.5
>>>>>
>>>>> But the CPU ratio akka2.2.3/akka 2.3.4 = 40/15 = 2.6.
>>>>>
>>>>> akka 2.2.3 use much more CPU for same throughput, does this mean akka
>>>>> 2.3.4 is more efficient? But why the message throughput drops on akka
>>>>> 2.3.4.
>>>>>
>>>>> I dumped the effective akka conf for 2.2.3 and 2.3.4 in the attachment.
>>>>>
>>>>>
>>>>> On Thursday, August 7, 2014 6:05:54 PM UTC+8, Sean Zhong wrote:
>>>>>>
>>>>>> Can it be the case that you have a lot of system message traffic
>>>>>>> between your systems? Do you have lots of remote deployed actors maybe?
>>>>>>
>>>>>>
>>>>>> All actors(4 source, 4 target) are created using remote actors.
>>>>>>
>>>>>>
>>>>>> On Thursday, August 7, 2014 5:45:34 PM UTC+8, drewhk wrote:
>>>>>>>
>>>>>>> Hi Sean,
>>>>>>>
>>>>>>> This is interesting, we actually measured increase with 2.3.4, in
>>>>>>> fact the relevant changes in that version were directly targeted to
>>>>>>> increasing performance somewhat. One important difference is that 2.3.4
>>>>>>> prioritizes internal Akka messages over user messages to improve
>>>>>>> stability.
>>>>>>> Can it be the case that you have a lot of system message traffic between
>>>>>>> your systems? Do you have lots of remote deployed actors maybe?
>>>>>>>
>>>>>>> -Endre
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 7, 2014 at 10:53 AM, Sean Zhong <cloc...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> When I upgrade from akka 2.2.3 to akka 2.3.4, I found the message
>>>>>>>> throughput drops about 30%.
>>>>>>>>
>>>>>>>> My benchmark looks like this:
>>>>>>>> 4 machines, each machine has 1 source actor and 1 target actor.
>>>>>>>> Each source actor will randomly deliver a 100 bytes messge at a time
>>>>>>>> to any
>>>>>>>> target actor.
>>>>>>>>
>>>>>>>> I use default configuration. Are there default configuration
>>>>>>>> changes in akka reference.conf which result in this behavior?
>>>>>>>>
>>>>>>>> --
>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c
>>>>>>>> urrent/additional/faq.html
>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou
>>>>>>>> p/akka-user
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Akka User List" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/
>>>> current/additional/faq.html
>>>> >>>>>>>>>> Search the archives: https://groups.google.com/
>>>> group/akka-user
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Akka User List" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to akka-user+...@googlegroups.com.
>>>> To post to this group, send email to akka...@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/akka-user.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> √
>>>
>> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
