[akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Hi All, 
Working on a project for a big client that want to switch from a monolithic 
infrastructure to a microservice one we are getting stuck in a big doubt.
We are making an infrastructure where each microservice are deployed in a 
different container on an ECS instance, than we have the first microservice 
that receive a message from the client via HTTP and this message triggers 
all the actors that start to send message one to each others.
We've created an Akka-Cluster and each microservice is a node of the 
cluster. Every Actor send a message to a Router 
(cluster-metrics-adaptive-group) that has his  routees.paths that points to 
the specific node.
Testing the performances of our system we realize that everything is too 
slow and there is not any multithreading.
How does multithreading work? Is our infrastructure too complex? Is there 
any best-practice we can implement to improve our system? 
Thanks all! 
Silvio

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Dear Konrad,
Thanks for your answer, our Actors didn't do anything that blocks the 
system.
One possible cause for our performance problem is that in our 
"cluster-metrics-adaptive-group" router have just one routees associated to 
one actor (We have one actor for microsistem right now). We think to 
increase the number of actor for each microsistem and consequencialy the 
routees lists, is this a possible solution?
If it is right to do so, there is a way to increase and decrease the group 
without going directly inside the code, and leaves akka do so ?
Thanks Silvio 

Il giorno lunedì 5 settembre 2016 11:26:58 UTC+2, Konrad Malawski ha 
scritto:
>
> This is not really enough information to say where or what the bottlenecks 
> are. If you need a full architecture overview that's a commercial thing we 
> can offer.
>
> Having that said, you did not mention what HTTP server you're using.
> Actors are simply multiplexed onto threads, so if you're doing blocking in 
> your actors you'll end up blocking the entire system - this could be one of 
> the reasons you're seeing bad perf and no parallelism.
> Another reason could be that you do all inside one actor or synchronously 
> in a thread, creating a bottleneck.
>
> Please explain a bit more on your setup and we could provide some more 
> helpful hints.
>
> Please also refer to the documentation when in doubt :)
>
> -- 
> Konrad `ktoso` Malawski
> Akka <http://akka.io> @ Lightbend <http://lightbend.com>
>
> On 5 September 2016 at 11:22:26, silvio poma (silvio...@gmail.com 
> ) wrote:
>
> Hi All,  
> Working on a project for a big client that want to switch from a 
> monolithic infrastructure to a microservice one we are getting stuck in a 
> big doubt.
> We are making an infrastructure where each microservice are deployed in a 
> different container on an ECS instance, than we have the first microservice 
> that receive a message from the client via HTTP and this message triggers 
> all the actors that start to send message one to each others.
> We've created an Akka-Cluster and each microservice is a node of the 
> cluster. Every Actor send a message to a Router 
> (cluster-metrics-adaptive-group) that has his  routees.paths that points to 
> the specific node.
> Testing the performances of our system we realize that everything is too 
> slow and there is not any multithreading.
> How does multithreading work? Is our infrastructure too complex? Is there 
> any best-practice we can implement to improve our system? 
> Thanks all! 
> Silvio
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+...@googlegroups.com .
> To post to this group, send email to akka...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
>>>>>>>>>>  Read the docs: http://akka.io/docs/
>>>>>>>>>>  Check the FAQ: 
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Dear Guido,
Thanks a lot for you answer. 
We have already improved our concurrency model, using multiple instances 
for each actor, that now work in parallel using multiple dispatcher.
Now we have not so much time to release our product to the client. 
What do you suggest to focus our efforts on? 
Which of this two arguments can effectively improve the system efficiency?

- Passing from Remote to Artery?
- Remove Java serializer and use another one?

Thanks,
Looking forward to hear you.
Silvio
  

Il giorno mercoledì 7 settembre 2016 11:16:08 UTC+2, Guido Medina ha 
scritto:
>
> Messages can be slow when being sent remotely from node A to node B,
> current Akka remote will give you a top of 100k msg/sec best scenario and 
> that depending on the message size,
> there is a new Akka remote (Akka artery which is a rewrite of Akka remote) 
> on the way so that shouldn't discourage you.
>
> Serialization is important, make sure you are not using the default Java 
> serialization, I suggest you to use Kryo for example.
>
> The fact that you are using actors doesn't really mean you are using 
> concurrency right, books like "Java concurrency in Practice" still apply, 
> you need to know how dispatchers work in Akka
> and how to spread your actors among dispatcher for specialized tasks, 
> also, you need to learn how to create several instances of the same actor 
> so that they act as workers,
> hence activating a proper level of concurrency.
>
> I think the main problem is that *-it is a common mistake by new Akka 
> users-* is to try to just throw actors at the problem and expect that it 
> will magically solve concurrency issues.
> You need to read the documentation and understand how actors, routers and 
> dispatchers can work together and how to combine them.
>
> If you are simply sending messages to one actor in a node you still have a 
> single threaded kind processor,
>
> I have tackled this kind of problem by creating several instances *-say N 
> x CPUs depending on the problem they are solving-*,
> send such list to a remote node and there create a dumb round-robin router 
> and use such router to send messages to such pool of actors,
> and then you will have some decent concurrency in that node, make sure you 
> use more than one dispatcher or else everything will be executed by the 
> same dispatcher,
> which translates to having a single thread pool executor doing everything.
>
> http://doc.akka.io/docs/akka/2.4.10/java/dispatchers.html
> http://doc.akka.io/docs/akka/2.4.10/java/routing.html
>
> HTH,
>
> Guido.
>
> On Monday, September 5, 2016 at 11:58:47 AM UTC+1, silvio poma wrote:
>>
>> Dear Konrad,
>> Thanks for your answer, our Actors didn't do anything that blocks the 
>> system.
>> One possible cause for our performance problem is that in our 
>> "cluster-metrics-adaptive-group" router have just one routees associated to 
>> one actor (We have one actor for microsistem right now). We think to 
>> increase the number of actor for each microsistem and consequencialy the 
>> routees lists, is this a possible solution?
>> If it is right to do so, there is a way to increase and decrease the 
>> group without going directly inside the code, and leaves akka do so ?
>> Thanks Silvio 
>>
>> Il giorno lunedì 5 settembre 2016 11:26:58 UTC+2, Konrad Malawski ha 
>> scritto:
>>>
>>> This is not really enough information to say where or what the 
>>> bottlenecks are. If you need a full architecture overview that's a 
>>> commercial thing we can offer.
>>>
>>> Having that said, you did not mention what HTTP server you're using.
>>> Actors are simply multiplexed onto threads, so if you're doing blocking 
>>> in your actors you'll end up blocking the entire system - this could be one 
>>> of the reasons you're seeing bad perf and no parallelism.
>>> Another reason could be that you do all inside one actor or 
>>> synchronously in a thread, creating a bottleneck.
>>>
>>> Please explain a bit more on your setup and we could provide some more 
>>> helpful hints.
>>>
>>> Please also refer to the documentation when in doubt :)
>>>
>>> -- 
>>> Konrad `ktoso` Malawski
>>> Akka <http://akka.io> @ Lightbend <http://lightbend.com>
>>>
>>> On 5 September 2016 at 11:22:26, silvio poma (silvio...@gmail.com) 
>>> wrote:
>>>
>>> Hi All,  
>>> Working on a project for a big client that want to switch from a 
>>> monolithic infrastructure to a microservice one we are getting stuck in a 
>>>

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Hi Guys, Thanks a lot everybody. Your tips were very useful and we are 
working on it.
We've already improved the router and the disptachers configuration 
achieving very good results.
Now we are trying to leave the java serialization and pass to another one 
more efficient, but we are having some troubles in this field.
Can some one point me any guides to follow? (Please notice that we are 
writing in Java and not in Scala)  
Thanks Guys,
Silvio

Il giorno mercoledì 7 settembre 2016 17:52:39 UTC+2, √ ha scritto:
>
> IMO Artery won't help much if using Java Serialization.
> Java Serialization is notoriously slow.
>
> On Wed, Sep 7, 2016 at 5:50 PM, Guido Medina  > wrote:
>
>> Artery is at  in milestone 3 (m3), you are welcome to use as I believe it 
>> is in an extremely good shape but I would pursue both,
>> as it will only require you a day of work (as a newbie) in configuration 
>> and dependencies, no code changes will be required:
>>
>>
>> http://search.maven.org/#artifactdetails%7Ccom.typesafe.akka%7Cakka-remote_2.11%7C2.4-ARTERY-M3%7Cjar
>>
>> As for dispatchers, be careful there, I didn't mean "one dispatcher per 
>> actor" but more like "one dispatcher per one or more types of actors"
>>
>> HTH,
>>
>> Guido.
>>
>>
>> On Wednesday, September 7, 2016 at 4:37:13 PM UTC+1, silvio poma wrote:
>>>
>>> Dear Guido,
>>> Thanks a lot for you answer. 
>>> We have already improved our concurrency model, using multiple instances 
>>> for each actor, that now work in parallel using multiple dispatcher.
>>> Now we have not so much time to release our product to the client. 
>>> What do you suggest to focus our efforts on? 
>>> Which of this two arguments can effectively improve the system 
>>> efficiency?
>>>
>>> - Passing from Remote to Artery?
>>> - Remove Java serializer and use another one?
>>>
>>> Thanks,
>>> Looking forward to hear you.
>>> Silvio
>>>   
>>>
>>> Il giorno mercoledì 7 settembre 2016 11:16:08 UTC+2, Guido Medina ha 
>>> scritto:
>>>>
>>>> Messages can be slow when being sent remotely from node A to node B,
>>>> current Akka remote will give you a top of 100k msg/sec best scenario 
>>>> and that depending on the message size,
>>>> there is a new Akka remote (Akka artery which is a rewrite of Akka 
>>>> remote) on the way so that shouldn't discourage you.
>>>>
>>>> Serialization is important, make sure you are not using the default 
>>>> Java serialization, I suggest you to use Kryo for example.
>>>>
>>>> The fact that you are using actors doesn't really mean you are using 
>>>> concurrency right, books like "Java concurrency in Practice" still apply, 
>>>> you need to know how dispatchers work in Akka
>>>> and how to spread your actors among dispatcher for specialized tasks, 
>>>> also, you need to learn how to create several instances of the same actor 
>>>> so that they act as workers,
>>>> hence activating a proper level of concurrency.
>>>>
>>>> I think the main problem is that *-it is a common mistake by new Akka 
>>>> users-* is to try to just throw actors at the problem and expect that 
>>>> it will magically solve concurrency issues.
>>>> You need to read the documentation and understand how actors, routers 
>>>> and dispatchers can work together and how to combine them.
>>>>
>>>> If you are simply sending messages to one actor in a node you still 
>>>> have a single threaded kind processor,
>>>>
>>>> I have tackled this kind of problem by creating several instances *-say 
>>>> N x CPUs depending on the problem they are solving-*,
>>>> send such list to a remote node and there create a dumb round-robin 
>>>> router and use such router to send messages to such pool of actors,
>>>> and then you will have some decent concurrency in that node, make sure 
>>>> you use more than one dispatcher or else everything will be executed by 
>>>> the 
>>>> same dispatcher,
>>>> which translates to having a single thread pool executor doing 
>>>> everything.
>>>>
>>>> http://doc.akka.io/docs/akka/2.4.10/java/dispatchers.html
>>>> http://doc.akka.io/docs/akka/2.4.10/java/routing.html
>>>>
>>>> HTH,
>>>>
>>>> Guido.
>>>

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Hi Guys, Thanks a lot everybody. Yours tips were very useful and we are 
working on it.
We've already improved the router and the disptachers configuration 
achieving very good results.
Now we are trying to leave the java serialization and pass to another one 
more efficient, but we are having some troubles in this field.
Can some one point me any guides to follow? (Please notice that we are 
writing in Java and not in Scala)  
Thanks Guys,
Silvio

Il giorno mercoledì 7 settembre 2016 17:52:39 UTC+2, √ ha scritto:
>
> IMO Artery won't help much if using Java Serialization.
> Java Serialization is notoriously slow.
>
> On Wed, Sep 7, 2016 at 5:50 PM, Guido Medina  > wrote:
>
>> Artery is at  in milestone 3 (m3), you are welcome to use as I believe it 
>> is in an extremely good shape but I would pursue both,
>> as it will only require you a day of work (as a newbie) in configuration 
>> and dependencies, no code changes will be required:
>>
>>
>> http://search.maven.org/#artifactdetails%7Ccom.typesafe.akka%7Cakka-remote_2.11%7C2.4-ARTERY-M3%7Cjar
>>
>> As for dispatchers, be careful there, I didn't mean "one dispatcher per 
>> actor" but more like "one dispatcher per one or more types of actors"
>>
>> HTH,
>>
>> Guido.
>>
>>
>> On Wednesday, September 7, 2016 at 4:37:13 PM UTC+1, silvio poma wrote:
>>>
>>> Dear Guido,
>>> Thanks a lot for you answer. 
>>> We have already improved our concurrency model, using multiple instances 
>>> for each actor, that now work in parallel using multiple dispatcher.
>>> Now we have not so much time to release our product to the client. 
>>> What do you suggest to focus our efforts on? 
>>> Which of this two arguments can effectively improve the system 
>>> efficiency?
>>>
>>> - Passing from Remote to Artery?
>>> - Remove Java serializer and use another one?
>>>
>>> Thanks,
>>> Looking forward to hear you.
>>> Silvio
>>>   
>>>
>>> Il giorno mercoledì 7 settembre 2016 11:16:08 UTC+2, Guido Medina ha 
>>> scritto:
>>>>
>>>> Messages can be slow when being sent remotely from node A to node B,
>>>> current Akka remote will give you a top of 100k msg/sec best scenario 
>>>> and that depending on the message size,
>>>> there is a new Akka remote (Akka artery which is a rewrite of Akka 
>>>> remote) on the way so that shouldn't discourage you.
>>>>
>>>> Serialization is important, make sure you are not using the default 
>>>> Java serialization, I suggest you to use Kryo for example.
>>>>
>>>> The fact that you are using actors doesn't really mean you are using 
>>>> concurrency right, books like "Java concurrency in Practice" still apply, 
>>>> you need to know how dispatchers work in Akka
>>>> and how to spread your actors among dispatcher for specialized tasks, 
>>>> also, you need to learn how to create several instances of the same actor 
>>>> so that they act as workers,
>>>> hence activating a proper level of concurrency.
>>>>
>>>> I think the main problem is that *-it is a common mistake by new Akka 
>>>> users-* is to try to just throw actors at the problem and expect that 
>>>> it will magically solve concurrency issues.
>>>> You need to read the documentation and understand how actors, routers 
>>>> and dispatchers can work together and how to combine them.
>>>>
>>>> If you are simply sending messages to one actor in a node you still 
>>>> have a single threaded kind processor,
>>>>
>>>> I have tackled this kind of problem by creating several instances *-say 
>>>> N x CPUs depending on the problem they are solving-*,
>>>> send such list to a remote node and there create a dumb round-robin 
>>>> router and use such router to send messages to such pool of actors,
>>>> and then you will have some decent concurrency in that node, make sure 
>>>> you use more than one dispatcher or else everything will be executed by 
>>>> the 
>>>> same dispatcher,
>>>> which translates to having a single thread pool executor doing 
>>>> everything.
>>>>
>>>> http://doc.akka.io/docs/akka/2.4.10/java/dispatchers.html
>>>> http://doc.akka.io/docs/akka/2.4.10/java/routing.html
>>>>
>>>> HTH,
>>>>
>>>> Guido.
>>>

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Right now one of the biggest and strange problem is that when arrive a 
message from the Actor A in the node 1 to the actor B in the node 2 the 
node 2 is immediately disassociated from the cluster. The message that the 
actor A sends to actor B is our custom class that we are serializing using 
kryo (as u suggested).

This is our config file: 

akka.extensions = 
["com.romix.akka.serialization.kryo.KryoSerializationExtension$"]

akka.actor.kryo  {

reference-enabled = true

   
implicit-registration-enabled = true

implicit-registration-logging = false


buffer-size = 4096

   
max-buffer-size = 1073741824


serializer-pool-size = 16

   
kryo-trace = false


kryo-reference-map = true

mappings {
"com.test.common.LoginRequest" = 20

   
}


classes = [
"com.test.common.LoginRequest"

]
}

#akka.actor.serialize-messages = on

akka.actor.serializers {
java = "akka.serialization.JavaSerializer"
kryo = "com.romix.akka.serialization.kryo.KryoSerializer"
}

akka.actor.serialization-bindings {

"com.test.common.LoginRequest" = kryo
} 

Il giorno venerdì 9 settembre 2016 15:32:32 UTC+2, Guido Medina ha scritto:
>
> What problems are you facing?
> I'm in my second project with Java 8 + Akka so maybe I can help.
>
> To integrate the akka-kryo extension all you need is the dependency and 
> few lines at the configuration you are using, no code changes are needed, 
> here is an example:
>
> akka {
>   // Log4j2 extension
>   loggers = ["de.heikoseeberger.akkalog4j.Log4jLogger"]
>   logging-filter = "de.heikoseeberger.akkalog4j.Log4jLoggingFilter"
>
>   // akka-kryo extension
>   extensions = 
> ["com.romix.akka.serialization.kryo.KryoSerializationExtension$"]
>
>   actor {
> provider = "akka.cluster.ClusterActorRefProvider"
> serializers.java = "com.romix.akka.serialization.kryo.KryoSerializer"
>
> serialization-bindings {
>   // Classes that don't implement serializable list here one by one
>   // to bind them to the Java serializer which you changed to Kryo.
>   "io.vertx.core.json.JsonObject" = java
> }
>
> // I don't like to have too many threads so for non-blocking stuff I 
> think N x CPU is enough where N = 1..4
> default-dispatcher {
>   type = Dispatcher
>   executor = "fork-join-executor"
>
>   fork-join-executor {
> parallelism-min = 4
> parallelism-factor = 1
> parallelism-max = 8
>   }
> }
>
> kryo {
>   kryo-reference-map = false
>   idstrategy = "automatic"
>   use-manifests = true
>   buffer-size = 1024
>   type = "nograph"
>
>   // To avoid serializing the class name list your remote message 
> classes here:
>   classes = [
> "com.company.class_1",
>
> "io.vertx.core.json.JsonObject",
>
> "java.util.ArrayList"
>   ]
> }
>   }
>
>   // My current remote configuration, enough threads (not too many) for a 
> cluster of up to 12 nodes
>   remote {
> log-remote-lifecycle-events = off
>
> netty.tcp {
>   port = 0
>
>   server-socket-worker-pool {
> pool-size-min = 4
> pool-size-factor = 1
> pool-size-max = 8
>   }
>
>   client-socket-worker-pool {
> pool-size-min = 4
> pool-size-factor = 1
> pool-size-max = 8
>   }
> }
>
> default-remote-dispatcher {
>   type = Dispatcher
>   executor = "fork-join-executor"
>
>   fork-join-executor {
> parallelism-min = 4
> parallelism-factor = 1
> parallelism-max = 8
>   }
> }
>   }
>
>   cluster {
> metrics.enabled = off
> jmx.enabled = off
>   }
> }
>
> HTH,
>
> Guido.
>
> On Friday, September 9, 2016 at 8:43:39 AM UTC+1, silvio poma wrote:
>>
>> Hi Guys, Thanks a lot everybody. Yours tips were very useful and we are 
>> working on it.
>> We've already improved the router and the disptachers configuration 
>> achieving very good results.
>> Now we are trying to leave the java serialization and pass to another one 
>> more efficient, but we are having some troubles in this field.
>> Can some one point me any guides to follow? (Please notice that we are 
>> writing in Java and not in Scala)  
>> Thanks Guys,
>> Silvio
>>
>> Il giorno mercoledì 7 settembre 2016 17:52:39 UTC+2, √ ha scritto:
>>>
&g

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Just to be clear, my class: com.test.common.LoginRequest  is formed of 4
strings and an actorRef
Thanks all

2016-09-09 16:34 GMT+02:00 silvio poma :

> Right now one of the biggest and strange problem is that when arrive a
> message from the Actor A in the node 1 to the actor B in the node 2 the
> node 2 is immediately disassociated from the cluster. The message that the
> actor A sends to actor B is our custom class that we are serializing using
> kryo (as u suggested).
>
> This is our config file:
>
> akka.extensions = ["com.romix.akka.serialization.kryo.
> KryoSerializationExtension$"]
>
> akka.actor.kryo  {
>
> reference-enabled = true
>
>
> implicit-registration-enabled = true
>
> implicit-registration-logging = false
>
>
> buffer-size = 4096
>
>
> max-buffer-size = 1073741824
>
>
> serializer-pool-size = 16
>
>
> kryo-trace = false
>
>
> kryo-reference-map = true
>
> mappings {
> "com.test.common.LoginRequest" = 20
>
>
> }
>
>
> classes = [
> "com.test.common.LoginRequest"
>
> ]
> }
>
> #akka.actor.serialize-messages = on
>
> akka.actor.serializers {
> java = "akka.serialization.JavaSerializer"
> kryo = "com.romix.akka.serialization.kryo.KryoSerializer"
> }
>
> akka.actor.serialization-bindings {
>
> "com.test.common.LoginRequest" = kryo
> }
>
> Il giorno venerdì 9 settembre 2016 15:32:32 UTC+2, Guido Medina ha scritto:
>>
>> What problems are you facing?
>> I'm in my second project with Java 8 + Akka so maybe I can help.
>>
>> To integrate the akka-kryo extension all you need is the dependency and
>> few lines at the configuration you are using, no code changes are needed,
>> here is an example:
>>
>> akka {
>>   // Log4j2 extension
>>   loggers = ["de.heikoseeberger.akkalog4j.Log4jLogger"]
>>   logging-filter = "de.heikoseeberger.akkalog4j.Log4jLoggingFilter"
>>
>>   // akka-kryo extension
>>   extensions = ["com.romix.akka.serialization
>> .kryo.KryoSerializationExtension$"]
>>
>>   actor {
>> provider = "akka.cluster.ClusterActorRefProvider"
>> serializers.java = "com.romix.akka.serialization.kryo.KryoSerializer"
>>
>> serialization-bindings {
>>   // Classes that don't implement serializable list here one by one
>>   // to bind them to the Java serializer which you changed to Kryo.
>>   "io.vertx.core.json.JsonObject" = java
>> }
>>
>> // I don't like to have too many threads so for non-blocking stuff I
>> think N x CPU is enough where N = 1..4
>> default-dispatcher {
>>   type = Dispatcher
>>   executor = "fork-join-executor"
>>
>>   fork-join-executor {
>> parallelism-min = 4
>> parallelism-factor = 1
>> parallelism-max = 8
>>   }
>> }
>>
>> kryo {
>>   kryo-reference-map = false
>>   idstrategy = "automatic"
>>   use-manifests = true
>>   buffer-size = 1024
>>   type = "nograph"
>>
>>   // To avoid serializing the class name list your remote message
>> classes here:
>>   classes = [
>> "com.company.class_1",
>>
>> "io.vertx.core.json.JsonObject",
>>
>> "java.util.ArrayList"
>>   ]
>> }
>>   }
>>
>>   // My current remote configuration, enough threads (not too many) for a
>> cluster of up to 12 nodes
>>   remote {
>> log-remote-lifecycle-events = off
>>
>> netty.tcp {
>>   port = 0
>>
>>   server-socket-worker-pool {
>> pool-size-min = 4
>> pool-size-factor = 1
>> pool-size-max = 8
>>   }
>>
>>   client-socket-worker-pool {
>> pool-size-min = 4
>> pool-size-factor = 1
>> pool-size-max = 8
>>   }
>> }
>>
>> default-remote-dispatcher {
>>   type = Dispatcher
>>   executor = "fork-join-executor"
>>
>>   fork-join-executor {
>> parallelism-min = 4
>> parallelism-factor = 1
>> parallelism-max = 8
>>   }
>> }
>>   }
>>
>>   cluster {
>> metrics.enabled = off
>> jmx.enabled = off
>>   }
>> }
>>

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Thanks guy, I eventually fixed all the configuration as you suggested and 
now everything is working.
I can see a big improvement in the average time of response of my service 
but the throughput is still the same of before.
Any other suggestions, my target is to have throughput as high as possible?

Thanks for yours tips!
Silvio 


Il giorno sabato 10 settembre 2016 12:11:35 UTC+2, Guido Medina ha scritto:
>
> Your contribution makes ID strategy automatic even better, you know 
> automatic = explicit + incremental right?
> So I think it would also work for your project, but I do understand why 
> you have chosen explicit as you want to fail if you introduce a new class 
> and forget to register it.
>
> I use automatic because I only register classes I need extremely optimize 
> in regards of message size over the wire which is important from the low 
> latency PoV,
> I believe your contribution should also work with automatic? do you happen 
> to know if it is tied to explicit or to the phase of registering listed 
> classes?
>
> If the latter then I believe it should also apply to automatic listed 
> classes to register.
>
> Guido.
>
> On Friday, September 9, 2016 at 8:36:54 PM UTC+1, Justin du coeur wrote:
>>
>> On Fri, Sep 9, 2016 at 2:51 PM, Guido Medina  wrote:
>>
>>> The problem with explicit is that there are so many "trait" and 
>>> "classes" not really registered which is why I use it,
>>> it is the newest ID strategy as it register things you might not care 
>>> but commonly used messages you register,
>>> that way you get performance for what is important and automatic 
>>> registration for other things.
>>>
>>
>> Yeah -- like I said, hardcore.  But my situation is extreme: I'm using 
>> Kryo for *persistent* serialization -- as far as I know, Querki may be the 
>> first project doing that in production.  That being the case, I care about 
>> the bytes taken up by FQDNs (and the time required to look them up, and the 
>> risks of things changing out from under me), so I've built out quite a 
>> bit 
>> 
>>  of 
>> infrastructure 
>> 
>>  
>> to make this work safely.
>>
>> (Yes, I know -- I'm insane.  But I like Kryo's attitude more than 
>> protobuf's from a code-cohesion POV, so I've invested the time to make it 
>> work solidly for Akka Persistence.)
>>
>> *Totally* understood about the sheer number of classes involved in making 
>> this work -- I spent quite a while wrestling with that.  But note that I 
>> contributed a new feature to the romix library 
>> 
>>  
>> to help with this.  The SubclassResolver allows you to register a single 
>> serializer for an entire type hierarchy, when that makes sense.  (Which it 
>> does for many of the common library types.)  I found that this cuts down on 
>> the number of "overhead" registrations required by a *huge* fraction -- you 
>> can suddenly deal with, eg, Set with a single registration instead of six.
>>
>> And yes, I need to find some time to write up a long blog post about all 
>> of this...
>>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

I have an http endpoint with a consumerActor (in Node 0) that read the msg 
from the camel msg and with it create our Message.class than it send to a 
router1 that forward to one of "N" Worker1 in Node 1, Worker1 make a query 
on Couchbase process the result and send the Message.class to to a router2 
that forward to one of "N" Worker2 in Node 2, Worker2 make a query on 
Couchbase process the result and send the Message.class to to a router3 
that forward to one of "N" Worker3 in Node 3, at the end Worker3 make a 
query on Couchbase process the result and send the http response. 

Il giorno martedì 13 settembre 2016 10:42:39 UTC+2, √ ha scritto:
>
> What is your bottleneck?
>
> On Tue, Sep 13, 2016 at 10:13 AM, silvio poma  > wrote:
>
>> Thanks guy, I eventually fixed all the configuration as you suggested and 
>> now everything is working.
>> I can see a big improvement in the average time of response of my service 
>> but the throughput is still the same of before.
>> Any other suggestions, my target is to have throughput as high as 
>> possible?
>>
>> Thanks for yours tips!
>> Silvio 
>>
>>
>> Il giorno sabato 10 settembre 2016 12:11:35 UTC+2, Guido Medina ha 
>> scritto:
>>>
>>> Your contribution makes ID strategy automatic even better, you know 
>>> automatic = explicit + incremental right?
>>> So I think it would also work for your project, but I do understand why 
>>> you have chosen explicit as you want to fail if you introduce a new class 
>>> and forget to register it.
>>>
>>> I use automatic because I only register classes I need extremely 
>>> optimize in regards of message size over the wire which is important from 
>>> the low latency PoV,
>>> I believe your contribution should also work with automatic? do you 
>>> happen to know if it is tied to explicit or to the phase of registering 
>>> listed classes?
>>>
>>> If the latter then I believe it should also apply to automatic listed 
>>> classes to register.
>>>
>>> Guido.
>>>
>>> On Friday, September 9, 2016 at 8:36:54 PM UTC+1, Justin du coeur wrote:
>>>>
>>>> On Fri, Sep 9, 2016 at 2:51 PM, Guido Medina  wrote:
>>>>
>>>>> The problem with explicit is that there are so many "trait" and 
>>>>> "classes" not really registered which is why I use it,
>>>>> it is the newest ID strategy as it register things you might not care 
>>>>> but commonly used messages you register,
>>>>> that way you get performance for what is important and automatic 
>>>>> registration for other things.
>>>>>
>>>>
>>>> Yeah -- like I said, hardcore.  But my situation is extreme: I'm using 
>>>> Kryo for *persistent* serialization -- as far as I know, Querki may be the 
>>>> first project doing that in production.  That being the case, I care about 
>>>> the bytes taken up by FQDNs (and the time required to look them up, and 
>>>> the 
>>>> risks of things changing out from under me), so I've built out quite a 
>>>> bit 
>>>> <https://github.com/jducoeur/Querki/tree/master/querki/scalajvm/app/querki/persistence>
>>>>  of 
>>>> infrastructure 
>>>> <https://github.com/jducoeur/Querki/tree/master/querki/scalajvm/test/querki/persistence>
>>>>  
>>>> to make this work safely.
>>>>
>>>> (Yes, I know -- I'm insane.  But I like Kryo's attitude more than 
>>>> protobuf's from a code-cohesion POV, so I've invested the time to make it 
>>>> work solidly for Akka Persistence.)
>>>>
>>>> *Totally* understood about the sheer number of classes involved in 
>>>> making this work -- I spent quite a while wrestling with that.  But note 
>>>> that I contributed a new feature to the romix library 
>>>> <https://github.com/romix/akka-kryo-serialization/commit/ffb2fec2dc9cfbca9c3503d6f32c6500cd1c40cd>
>>>>  
>>>> to help with this.  The SubclassResolver allows you to register a single 
>>>> serializer for an entire type hierarchy, when that makes sense.  (Which it 
>>>> does for many of the common library types.)  I found that this cuts down 
>>>> on 
>>>> the number of "overhead" registrations required by a *huge* fraction -- 
>>>> you 
>>>> can suddenly deal with, eg, Set with a single registration in

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Hi Guido, can you suggest me some guide that explains the configuration 
changes that need to be done?
I've seen that with the M4 it was included the configuration for for usage 
with Docker, that is what i need.

Thanks Silvio

Il giorno mercoledì 7 settembre 2016 17:50:44 UTC+2, Guido Medina ha 
scritto:
>
> Artery is at  in milestone 3 (m3), you are welcome to use as I believe it 
> is in an extremely good shape but I would pursue both,
> as it will only require you a day of work (as a newbie) in configuration 
> and dependencies, no code changes will be required:
>
>
> http://search.maven.org/#artifactdetails%7Ccom.typesafe.akka%7Cakka-remote_2.11%7C2.4-ARTERY-M3%7Cjar
>
> As for dispatchers, be careful there, I didn't mean "one dispatcher per 
> actor" but more like "one dispatcher per one or more types of actors"
>
> HTH,
>
> Guido.
>
> On Wednesday, September 7, 2016 at 4:37:13 PM UTC+1, silvio poma wrote:
>>
>> Dear Guido,
>> Thanks a lot for you answer. 
>> We have already improved our concurrency model, using multiple instances 
>> for each actor, that now work in parallel using multiple dispatcher.
>> Now we have not so much time to release our product to the client. 
>> What do you suggest to focus our efforts on? 
>> Which of this two arguments can effectively improve the system efficiency?
>>
>> - Passing from Remote to Artery?
>> - Remove Java serializer and use another one?
>>
>> Thanks,
>> Looking forward to hear you.
>> Silvio
>>   
>>
>> Il giorno mercoledì 7 settembre 2016 11:16:08 UTC+2, Guido Medina ha 
>> scritto:
>>>
>>> Messages can be slow when being sent remotely from node A to node B,
>>> current Akka remote will give you a top of 100k msg/sec best scenario 
>>> and that depending on the message size,
>>> there is a new Akka remote (Akka artery which is a rewrite of Akka 
>>> remote) on the way so that shouldn't discourage you.
>>>
>>> Serialization is important, make sure you are not using the default Java 
>>> serialization, I suggest you to use Kryo for example.
>>>
>>> The fact that you are using actors doesn't really mean you are using 
>>> concurrency right, books like "Java concurrency in Practice" still apply, 
>>> you need to know how dispatchers work in Akka
>>> and how to spread your actors among dispatcher for specialized tasks, 
>>> also, you need to learn how to create several instances of the same actor 
>>> so that they act as workers,
>>> hence activating a proper level of concurrency.
>>>
>>> I think the main problem is that *-it is a common mistake by new Akka 
>>> users-* is to try to just throw actors at the problem and expect that 
>>> it will magically solve concurrency issues.
>>> You need to read the documentation and understand how actors, routers 
>>> and dispatchers can work together and how to combine them.
>>>
>>> If you are simply sending messages to one actor in a node you still have 
>>> a single threaded kind processor,
>>>
>>> I have tackled this kind of problem by creating several instances *-say 
>>> N x CPUs depending on the problem they are solving-*,
>>> send such list to a remote node and there create a dumb round-robin 
>>> router and use such router to send messages to such pool of actors,
>>> and then you will have some decent concurrency in that node, make sure 
>>> you use more than one dispatcher or else everything will be executed by the 
>>> same dispatcher,
>>> which translates to having a single thread pool executor doing 
>>> everything.
>>>
>>> http://doc.akka.io/docs/akka/2.4.10/java/dispatchers.html
>>> http://doc.akka.io/docs/akka/2.4.10/java/routing.html
>>>
>>> HTH,
>>>
>>> Guido.
>>>
>>> On Monday, September 5, 2016 at 11:58:47 AM UTC+1, silvio poma wrote:
>>>>
>>>> Dear Konrad,
>>>> Thanks for your answer, our Actors didn't do anything that blocks the 
>>>> system.
>>>> One possible cause for our performance problem is that in our 
>>>> "cluster-metrics-adaptive-group" router have just one routees associated 
>>>> to 
>>>> one actor (We have one actor for microsistem right now). We think to 
>>>> increase the number of actor for each microsistem and consequencialy the 
>>>> routees lists, is this a possible solution?
>>>> If it is right to do so, there is a way to

Re: [akka-user] Akka Multithreading on Akka-Cluster

[silvio poma]

Hi Guido,
Following the release notes i create a config file like this:


"remote": {
 "adapters": {
 "gremlin": "akka.remote.transport.FailureInjectorProvider",
 "trttl": "akka.remote.transport.ThrottlerProvider"
 },
 "artery": {
 "bind-hostname": "127.0.1.1",
 "enabled": "on",
 "hostname": "127.0.1.1",
 "tcp": {
 "bind-port": 2551,
 "port": 2551
 }
 },
 "backoff-interval": "5 ms",
 "backoff-remote-dispatcher": {
 "executor": "fork-join-executor",
 "fork-join-executor": {
 "parallelism-max": 2,
 "parallelism-min": 2
 },
 "type": "Dispatcher"
 },
 "command-ack-timeout": "30 s",
 "default-remote-dispatcher": {
 "executor": "fork-join-executor",
 "fork-join-executor": {
 "parallelism-max": 2,
 "parallelism-min": 2
 },
 "type": "Dispatcher"
 },
 "enabled-transports": ["akka.remote.netty.tcp"],
 "flush-wait-on-shutdown": "2 s",
 "gremlin": {
 "debug": "off"
 },
 "initial-system-message-delivery-timeout": "3 m",
 "log-buffer-size-exceeding": 5,
 "log-frame-size-exceeding": "off",
 "log-received-messages": "off",
 "log-remote-lifecycle-events": "off",
 "log-sent-messages": "off",
 "netty": {
 "ssl": {
 "applied-adapters": [],
 "backlog": 4096,
 "bind-hostname": "",
 "bind-port": "",
 "client-socket-worker-pool": {
 "pool-size-factor": 1,
 "pool-size-max": 2,
 "pool-size-min": 2
 },
 "connection-timeout": "15 s",
 "enable-ssl": true,
 "hostname": "",
 "maximum-frame-size": "128000b",
 "port": 2552,
 "receive-buffer-size": "256000b",
 "security": {
 "enabled-algorithms": ["TLS_RSA_WITH_AES_128_CBC_SHA"],
 "key-password": "changeme",
 "key-store": "keystore",
 "key-store-password": "changeme",
 "protocol": "TLSv1",
 "random-number-generator": "",
 "trust-store": "truststore",
 "trust-store-password": "changeme"
 },
 "send-buffer-size": "256000b",
 "server-socket-worker-pool": {
 "pool-size-factor": 1,
 "pool-size-max": 2,
 "pool-size-min": 2
 },
 "tcp-keepalive": "on",
 "tcp-nodelay": "on",
 "tcp-reuse-addr": "off-for-windows",
 "transport-class": "akka.remote.transport.netty.NettyTransport",
 "transport-protocol": "tcp",
 "use-dispatcher-for-io": "",
 "write-buffer-high-water-mark": "0b",
 "write-buffer-low-water-mark": "0b"
 },
 "tcp": {
 "applied-adapters": [],
 "backlog": 4096,
 "bind-hostname": "",
 "bind-port": "",
 "client-socket-worker-pool": {
 "pool-size-factor": 1,
 "pool-size-max": 2,
 "pool-size-min": 2
 },
 "connection-timeout": "15 s",
 "enable-ssl": false,
 "hostname": "",
 "maximum-frame-size": "128000b",
 "port": 2552,
 "receive-buffer-size": "256000b",
 "send-buffer-size": "256000b",
 "server-socket-worker-pool": {
 "pool-size-factor": 1,
 "pool-size-max": 2,
 "pool-size-min": 2
 },
 "tcp-keepalive": "on",
 "tcp-nodelay": "on",
 "tcp-reuse-addr": "off-for-windows",
 "transport-class": "akka.remote.transport.netty.NettyTransport",
 "transport-protocol": "tcp",
 "use-dispatcher-for-io": "",
 "write-buffer-high-water-mark": "0b",
 "write-buffer-low-water-mark": "0b"
 },
 "udp": {
 "applied-adapters": [],
 "backlog": 4096,
 "bind-hostname": "",
 "bind-port": "",
 "client-socket-worker-pool": {
 "pool-size-factor": 1,
 "pool-size-max": 2,
 "pool-size-min": 2
 },
 "connection-timeout": "15 s",
 "enable-ssl": false,
 "hostname": "",
 "maximum-frame-size": "128000b",
 "port": 2552,
 "receive-buffer-size": "256000b",
 "send-buffer-size": "256000b",
 "server-socket-worker-pool": {
 "pool-size-factor": 1,
 "pool-size-max": 2,
 "pool-size-min": 2
 },
 "tcp-keepalive": "on",
 "tcp-nodelay": "on",
 "tcp-reuse-addr": "off-for-windows",
 "transport-class": "akka.remote.transport.netty.NettyTransport",
 "transport-protocol": "udp",
 "use-dispatcher-for-io": "",
 "write-buffer-high-water-mark": "0b",
 "write-buffer-low-water-mark": "0b"
 }
 },



and i had this error:

No transport is loaded for protocol: [artery], available protocols: [akka.
tcp]
akka.remote.RemoteTransportException: No transport is loaded for protocol: [
artery], available protocols: [akka.tcp]


Anyway yes i measured the CouchDB query it's go from 30 to 300 ms more or 
less. And i implemented it with future so it should be a blocking 
operation, am i right?

Thanks!

Il giorno mercoledì 14 settembre 2016 10:31:43 UTC+2, Guido Medina ha 
scritto:
>
> What you have seen for Akka Artery M4 is what is available at the moment,
> I did test it when it was M2 but I haven't done anything with M4 yet,
> for M4 I imagine the same applies as it was for M4, the release notes is 
> the guide.
>
> Have you measured your application like you were suggested?
> I imagine you can at least log how many ms takes to query from CouchDB?
> I also imagine at Node 1 your actors querying from CouchDB are running on 
> a separate dispatcher because any DB operation is considered to be a 
> blocking operation?
>
> I believe you need to figure out where is your bottleneck first.
>
> HTH,
>
> Guido.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>