Ok. In our case the ack is per message... anyway I could expect a very good improvement in number of transactions upgrading to Artery. We don't need the factor 30x you have. If we have a factor 2 or 3 it would be very welcome.
Thanks for the info. This numbers could help me to push it into a near sprint. Best regards Patrik. On Mon, Jul 4, 2016 at 9:18 AM, Patrik Nordwall <patrik.nordw...@gmail.com> wrote: > > > On Mon, Jul 4, 2016 at 9:13 AM, Eduardo Fernandes <edu...@gmail.com> > wrote: > >> Wow!!! I have to try the new remoting! >> >> The measurement is quite basic: I have a pool of threads and I just call >> the sends() and wait for the response and execute a new send. Nothing >> particular. I made a simple test with the old remote and my numbers were >> also around 25.000 msg/s, so we agree on that. On top of the old remote, on >> my local machine (4 cores) I set the client and the server and I get around >> 300.000 msg/s (really are 600.000 because we have cli->server and other >> server->client response). So I could expect a very good enhancement if we >> move to Artery. >> >> The limitation in my machine is CPU, not network, of course, because all >> operations go through the loopback virtual card, which is a lot faster than >> the real card. The CPU is high because of serialization. >> >> In your test of 700.000 does this include a response from the server (or >> remote peer or whatever)? >> > > No, that's one-way between two JVMs. Flow control is handled by one ack > per batch of 1000 messages (several of these batches in flight). > > >> >> Thanks. >> >> >> On Mon, Jul 4, 2016 at 9:03 AM, Patrik Nordwall < >> patrik.nordw...@gmail.com> wrote: >> >>> Then I question how you measure this in your benchmark. With old >>> (current) remoting I have observed max throughput of 25.000 msg/s (one way) >>> using this test: >>> https://github.com/akka/akka/blob/artery-dev/akka-remote-tests/src/multi-jvm/scala/akka/remote/artery/MaxThroughputSpec.scala >>> >>> With new remoting (Artery) that test performs around 700.000 msg/s on my >>> local machine. >>> >>> >>> On Mon, Jul 4, 2016 at 8:54 AM, Eduardo Fernandes <edu...@gmail.com> >>> wrote: >>> >>>> Yes... 1.000.000 messages over the network (1.000.000 sent and >>>> 1.000.000 of ack's with the operation state, in this case a single echo). >>>> Sorry, I was not precise on that. We call it transaction becase we make a >>>> kind of commit in ram but in this case it is not relevant. >>>> >>>> On Mon, Jul 4, 2016 at 8:51 AM, Patrik Nordwall < >>>> patrik.nordw...@gmail.com> wrote: >>>> >>>>> Are you talking about 1.000.000 (500.000) messages/s over the network >>>>> or what is your definition of transaction? >>>>> >>>>> On Mon, Jul 4, 2016 at 8:47 AM, Eduardo Fernandes <edu...@gmail.com> >>>>> wrote: >>>>> >>>>>> Many thanks Patrik. I'll share it with our dev team. I've read it >>>>>> when we have indeed to change our code a bit. If we can distribute the >>>>>> serialization across the processors I'm pretty sure we could achieve >>>>>> around >>>>>> 1.000.000 transactions/s in an 8 cores machine with 4 or 5 actors on each >>>>>> node. Right now I have around 500.000 with 2 actors but increasing the >>>>>> number of actors is not increasing the numbers as we should expect. The >>>>>> network bandwidth is not the problem and I have around 48% of CPU in idle >>>>>> state. >>>>>> >>>>>> Regards and thanks again for the info. >>>>>> >>>>>> On Mon, Jul 4, 2016 at 7:56 AM, Patrik Nordwall < >>>>>> patrik.nordw...@gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 4, 2016 at 12:09 AM, Eduardo Fernandes <edu...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Ok. I'll manage to change from 2.10 to 2.11. Yes, my project is in >>>>>>>> Java. We're using Java 8 but the transition to 2.4 is not direct since >>>>>>>> we've overriden some behaviors in 2.3 which changed in 2.4 ( for ex. >>>>>>>> AllocationStrategy). I've tried to change to 2.4 and I got many errors. >>>>>>>> >>>>>>> >>>>>>> Updating to 2.4 is a good idea, since OSS version of 2.3 is >>>>>>> end-of-life. You find the migration guide here: >>>>>>> http://doc.akka.io/docs/akka/2.4.7/project/migration-guide-2.3.x-2.4.x.html >>>>>>> >>>>>>> I would like to clarify one thing regarding the release of the new >>>>>>> remoting (Artery). We are still developing it and we are releasing >>>>>>> development milestones that you can try out. M3 to be released end of >>>>>>> next >>>>>>> week according to the plan >>>>>>> <https://github.com/akka/akka-meta/issues/22>. It will be merged >>>>>>> back to 2.4 and released in a 2.4.x version, but that will not happen >>>>>>> next >>>>>>> week. >>>>>>> >>>>>>> /Patrik >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Registering classes implicitly is not possible since we have many >>>>>>>> serialization asymmetries so I have to list the classes explicitly in >>>>>>>> configuration, as you've said. I enabled kryo traces to print out >>>>>>>> automatically registered classed, which is very helpful. >>>>>>>> >>>>>>>> I'm working on it. >>>>>>>> >>>>>>>> Thanks again for you patience. >>>>>>>> >>>>>>>> /Eduardo >>>>>>>> >>>>>>>> On Sun, Jul 3, 2016 at 11:53 PM, Guido Medina <oxyg...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Questions: >>>>>>>>> >>>>>>>>> - Why not just replace 2.10 with 2.11? From the Java >>>>>>>>> perspective it should be transparent and irrelevant to your >>>>>>>>> project *-if >>>>>>>>> I understood correctly, your project is in Java-* >>>>>>>>> - Are you using Java 8? If just why not just go Akka 2.4.x? >>>>>>>>> >>>>>>>>> Next week Akka 2.4.8+ (if sprint goes well I think) will get a new >>>>>>>>> juice, akka-artery which will replace the current Netty basically >>>>>>>>> giving >>>>>>>>> you a much faster remote component. >>>>>>>>> The automatic ID strategy is only available for akka-kryo 0.4.1+ >>>>>>>>> which uses 2.11, automatic was the name we came up for default + >>>>>>>>> incremental, basically to allow the developer to: >>>>>>>>> >>>>>>>>> - Register classes explicitly, for the ones registered manual >>>>>>>>> performance will be better. >>>>>>>>> - Register classes automatically not listed. >>>>>>>>> >>>>>>>>> HTH, >>>>>>>>> >>>>>>>>> Guido. >>>>>>>>> >>>>>>>>> On Sunday, July 3, 2016 at 10:39:50 PM UTC+1, Eduardo Fernandes >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi All. >>>>>>>>>> >>>>>>>>>> I'm using akka-kryo-serialization_2.10 with incremental strategy. >>>>>>>>>> I'm using 2.10 for everything so maybe I should update in a near >>>>>>>>>> future. >>>>>>>>>> >>>>>>>>>> I'm getting the error below. >>>>>>>>>> >>>>>>>>>> Anyway, after tunning kryo to work with my project, I never will >>>>>>>>>> get more cpu's working in my bench so I'm afraid that I'll reach kryo >>>>>>>>>> limits quite soon. >>>>>>>>>> >>>>>>>>>> To fully adapt kryo to our project maybe I need to include it in >>>>>>>>>> a sprint. I have a considerable number of serializable classes. >>>>>>>>>> >>>>>>>>>> Thanks again for your time. >>>>>>>>>> >>>>>>>>>> --------------------- >>>>>>>>>> >>>>>>>>>> Serialization trace: >>>>>>>>>> callbacks >>>>>>>>>> (com.appgree.core.objectserver.providers.akka.client.ResponseImpl) >>>>>>>>>> response >>>>>>>>>> (com.appgree.core.objectserver.providers.akka.commands.CommandRequest) >>>>>>>>>> commands >>>>>>>>>> (com.appgree.core.objectserver.providers.akka.commands.CommandBulkRequest) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:682) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:737) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:368) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289) >>>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:682) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:660) >>>>>>>>>> at >>>>>>>>>> com.romix.akka.serialization.kryo.KryoBasedSerializer.fromBinary(KryoSerializer.scala:397) >>>>>>>>>> at >>>>>>>>>> com.romix.akka.serialization.kryo.KryoSerializer.fromBinary(KryoSerializer.scala:239) >>>>>>>>>> at >>>>>>>>>> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104) >>>>>>>>>> at scala.util.Try$.apply(Try.scala:161) >>>>>>>>>> at >>>>>>>>>> akka.serialization.Serialization.deserialize(Serialization.scala:98) >>>>>>>>>> at >>>>>>>>>> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23) >>>>>>>>>> at >>>>>>>>>> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58) >>>>>>>>>> at >>>>>>>>>> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58) >>>>>>>>>> at >>>>>>>>>> akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76) >>>>>>>>>> at >>>>>>>>>> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:929) >>>>>>>>>> at akka.actor.Actor$class.aroundReceive(Actor.scala:467) >>>>>>>>>> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:405) >>>>>>>>>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>>>>>>>>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>>>>>>>>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>>>>>>>>> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>>>>>>>>> at >>>>>>>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) >>>>>>>>>> at >>>>>>>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>>>>>>>>> at >>>>>>>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>>>>>>>>> at >>>>>>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>>>>>>>>> at >>>>>>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>>>>>>>>> Caused by: java.lang.NullPointerException >>>>>>>>>> at java.util.ArrayList.ensureExplicitCapacity(Unknown Source) >>>>>>>>>> at java.util.ArrayList.ensureCapacity(Unknown Source) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:96) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22) >>>>>>>>>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:682) >>>>>>>>>> at >>>>>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) >>>>>>>>>> ... 32 more >>>>>>>>>> >>>>>>>>>> On Sun, Jul 3, 2016 at 3:43 PM, Eduardo Fernandes < >>>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Ok. Thanks again. >>>>>>>>>>> >>>>>>>>>>> I'll give it a try. I have to adapt our code a bit. >>>>>>>>>>> >>>>>>>>>>> Thanks for your time on this. >>>>>>>>>>> >>>>>>>>>>> El 3 jul 2016, a las 15:14, Guido Medina <oxy...@gmail.com> >>>>>>>>>>> escribió: >>>>>>>>>>> >>>>>>>>>>> *Correction:* >>>>>>>>>>> >>>>>>>>>>> - Kryo offers a pool that you can use if you use Kryo >>>>>>>>>>> directly, kind of the recommended way of using it. >>>>>>>>>>> - Akka Kryo uses a similar model giving you the choice of >>>>>>>>>>> using your own queue implementation for its internal pool of >>>>>>>>>>> instances, >>>>>>>>>>> each instance is an Akka serializer with its Kryo plus other >>>>>>>>>>> things. >>>>>>>>>>> >>>>>>>>>>> You can see the relevant Akka Kryo code for the pool in this >>>>>>>>>>> commit/diff: >>>>>>>>>>> https://github.com/romix/akka-kryo-serialization/commit/045cd27dfd01c2c41ab7c64bf6e25a63b3fd8e42 >>>>>>>>>>> >>>>>>>>>>> Guido. >>>>>>>>>>> >>>>>>>>>>> On Sunday, July 3, 2016 at 1:47:45 PM UTC+1, Guido Medina wrote: >>>>>>>>>>>> >>>>>>>>>>>> Kryo and Akka Kryo don't use any threads, the remote-dispatcher >>>>>>>>>>>> is the one doing the work, >>>>>>>>>>>> and Netty threads sending/receiving the bytes from/to Akka >>>>>>>>>>>> remote dispatcher as far as I can tell. >>>>>>>>>>>> >>>>>>>>>>>> Both Kryo and Akka Kryo have a pool with Kryo instances which >>>>>>>>>>>> also happens to re-use the byte buffer making it very efficient in >>>>>>>>>>>> regards >>>>>>>>>>>> of GC. >>>>>>>>>>>> >>>>>>>>>>>> Kryo pool by default uses a Java ConcurrentQueue (I contributed >>>>>>>>>>>> some of it to Akka Kryo extension) which you can change. >>>>>>>>>>>> Read the section, KryoQueueBuilder: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/romix/akka-kryo-serialization#kryo-queue-builder-examples >>>>>>>>>>>> >>>>>>>>>>>> which will also improve even further the GC collection. >>>>>>>>>>>> >>>>>>>>>>>> HTH, >>>>>>>>>>>> >>>>>>>>>>>> Guido. >>>>>>>>>>>> >>>>>>>>>>>> On Sunday, July 3, 2016 at 1:28:46 PM UTC+1, Eduardo Fernandes >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Quite clean. Many thanks Guido! I'll try it out. I have to >>>>>>>>>>>>> inventory my classes so it will take a while. I hope I'll have >>>>>>>>>>>>> some numbers >>>>>>>>>>>>> this night. >>>>>>>>>>>>> >>>>>>>>>>>>> Anyway, kryo will scale vertically, I suppose. Will kryo use >>>>>>>>>>>>> more threads than I'm using right now? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks again for your help and knowledge. >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Jul 3, 2016 at 2:13 PM, Guido Medina <oxy...@gmail.com >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I meant to say *"anything that implements Serializable"* >>>>>>>>>>>>>> The classes list is important as Kryo won't write class names >>>>>>>>>>>>>> on the messages but IDs of the classes: >>>>>>>>>>>>>> >>>>>>>>>>>>>> classes = [ >>>>>>>>>>>>>> "com.mypackage.Class1", >>>>>>>>>>>>>> "com.mypackage.Class2" >>>>>>>>>>>>>> ] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Suffice to say every node of the cluster must have the same >>>>>>>>>>>>>> IDs so that's some sort of configuration you agree upon. >>>>>>>>>>>>>> I find convenient to have a package in a "common" jar project >>>>>>>>>>>>>> with all the classes that I'm going to share (Serialize-ables) >>>>>>>>>>>>>> and a "common.conf" with Akka configuration, as I can build >>>>>>>>>>>>>> my final configuration by putting together Akka configurations, >>>>>>>>>>>>>> even using place holders, like this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> config = parseFile(new File(configPath)).withFallback( >>>>>>>>>>>>>> parseResources("common.conf"))...resolve(); >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can use that fallback call as many times as you wish and >>>>>>>>>>>>>> it won't to resolve place holders until you don't call resolve() >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTH, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Guido. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sunday, July 3, 2016 at 12:57:21 PM UTC+1, Guido Medina >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have mimic-ed your configuration and corrected some >>>>>>>>>>>>>>> errors, also added Kryo if you want to give it a chance with the >>>>>>>>>>>>>>> configuration I believe will do best. >>>>>>>>>>>>>>> I default the "java" serializer to Kryo, that way, >>>>>>>>>>>>>>> everything that inherits "Serializable" will use Kryo, also, I >>>>>>>>>>>>>>> list every >>>>>>>>>>>>>>> class that I care (performance wise) under Kryo list. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hope this give you better result, also, don't underestimate >>>>>>>>>>>>>>> the the default mailbox you have commented out: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> akka { >>>>>>>>>>>>>>> extensions = >>>>>>>>>>>>>>> ["com.romix.akka.serialization.kryo.KryoSerializationExtension$"] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> actor { >>>>>>>>>>>>>>> provider = "akka.cluster.ClusterActorRefProvider" >>>>>>>>>>>>>>> serializers.java = >>>>>>>>>>>>>>> "com.romix.akka.serialization.kryo.KryoSerializer" >>>>>>>>>>>>>>> default-mailbox.mailbox-type = >>>>>>>>>>>>>>> "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> default-dispatcher { >>>>>>>>>>>>>>> type = Dispatcher >>>>>>>>>>>>>>> executor = "fork-join-executor" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> fork-join-executor { >>>>>>>>>>>>>>> parallelism-min = 4 >>>>>>>>>>>>>>> parallelism-factor = 1 >>>>>>>>>>>>>>> parallelism-max = 8 >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kryo { >>>>>>>>>>>>>>> kryo-reference-map = false >>>>>>>>>>>>>>> idstrategy = "automatic" >>>>>>>>>>>>>>> use-manifests = true >>>>>>>>>>>>>>> buffer-size = 1024 >>>>>>>>>>>>>>> type = "nograph" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> classes = [ >>>>>>>>>>>>>>> "com.mypackage.Class1", >>>>>>>>>>>>>>> "com.mypackage.Class2" >>>>>>>>>>>>>>> ] >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> remote { >>>>>>>>>>>>>>> log-remote-lifecycle-events = off >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> netty.tcp { >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> server-socket-worker-pool { >>>>>>>>>>>>>>> pool-size-min = 4 >>>>>>>>>>>>>>> pool-size-factor = 1 >>>>>>>>>>>>>>> pool-size-max = 8 >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> client-socket-worker-pool { >>>>>>>>>>>>>>> pool-size-min = 4 >>>>>>>>>>>>>>> pool-size-factor = 1 >>>>>>>>>>>>>>> pool-size-max = 8 >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> default-remote-dispatcher { >>>>>>>>>>>>>>> type = Dispatcher >>>>>>>>>>>>>>> executor = "fork-join-executor" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> fork-join-executor { >>>>>>>>>>>>>>> parallelism-min = 4 >>>>>>>>>>>>>>> parallelism-factor = 1 >>>>>>>>>>>>>>> parallelism-max = 8 >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cluster { >>>>>>>>>>>>>>> metrics.enabled = off >>>>>>>>>>>>>>> jmx.enabled = off >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sunday, July 3, 2016 at 12:45:57 PM UTC+1, Eduardo >>>>>>>>>>>>>>> Fernandes wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm using the configuration below, following Guido >>>>>>>>>>>>>>>> suggestions. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Anyway, everything behaves like Akka/netty add more >>>>>>>>>>>>>>>> communication threads only if a add more endpoints. Adding >>>>>>>>>>>>>>>> more actores >>>>>>>>>>>>>>>> talking from host 1 to host 2 is not improving throughput, >>>>>>>>>>>>>>>> despite of the >>>>>>>>>>>>>>>> available idle cpu's. Does this make sense? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Of course if my processing were not only dedicated to echo >>>>>>>>>>>>>>>> back some characters the processing cpu would hide the >>>>>>>>>>>>>>>> serialization >>>>>>>>>>>>>>>> performance. It just for find the maximum throughput (net >>>>>>>>>>>>>>>> performance) of a >>>>>>>>>>>>>>>> couple of actorsystem's. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My topology is (attached file). Each client process has >>>>>>>>>>>>>>>> 10.000 internal sync clients and uses a single thread to >>>>>>>>>>>>>>>> dispatch messages. >>>>>>>>>>>>>>>> A single client reaches around 450.000/s and two reaches >>>>>>>>>>>>>>>> around 525.000/s. >>>>>>>>>>>>>>>> Each client talks to a particular actor in host 2 which talks >>>>>>>>>>>>>>>> to a >>>>>>>>>>>>>>>> particular actor in host 3. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Again, many thanks for your help. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Again, many thanks for your help. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> actor { >>>>>>>>>>>>>>>> provider = "akka.cluster.ClusterActorRefProvider" >>>>>>>>>>>>>>>> default-dispatcher { >>>>>>>>>>>>>>>> throughput = 1024 >>>>>>>>>>>>>>>> fork-join-executor { >>>>>>>>>>>>>>>> parallelism-min = 6 >>>>>>>>>>>>>>>> parallelism-factor = 1 >>>>>>>>>>>>>>>> parallelism-max = 8 >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # default-mailbox { >>>>>>>>>>>>>>>> # mailbox-type = >>>>>>>>>>>>>>>> "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" >>>>>>>>>>>>>>>> # } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> remote { >>>>>>>>>>>>>>>> log-remote-lifecycle-events = off >>>>>>>>>>>>>>>> netty.tcp { >>>>>>>>>>>>>>>> tcp-nodelay = on >>>>>>>>>>>>>>>> write-buffer-high-water-mark = 40000000b >>>>>>>>>>>>>>>> write-buffer-low-water-mark = 0b >>>>>>>>>>>>>>>> send-buffer-size = 40000000b >>>>>>>>>>>>>>>> receive-buffer-size = 40000000b >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> server-socket-worker-pool { >>>>>>>>>>>>>>>> pool-size-min = 4 >>>>>>>>>>>>>>>> pool-size-factor = 1 >>>>>>>>>>>>>>>> pool-size-max = 8 >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> client-socket-worker-pool { >>>>>>>>>>>>>>>> pool-size-min = 4 >>>>>>>>>>>>>>>> pool-size-factor = 1 >>>>>>>>>>>>>>>> pool-size-max = 8 >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sun, Jul 3, 2016 at 1:05 PM, Roland Kuhn < >>>>>>>>>>>>>>>> goo...@rkuhn.info> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Doesn't the classical remoting perform serialization >>>>>>>>>>>>>>>>> within the single actor responsible for each connection? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 03 Jul 2016, at 12:57, Viktor Klang < >>>>>>>>>>>>>>>>> viktor...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Eduardo, are you sure that there aren't any >>>>>>>>>>>>>>>>> synchronized-blocks or locks used? (i.e. is this a contention >>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>> rather than a paralellization problem?) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sun, Jul 3, 2016 at 12:28 PM, Eduardo Fernandes < >>>>>>>>>>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ups... sorry for misunderstanding your question. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> My principal problem is not the overhead itself. My >>>>>>>>>>>>>>>>>> problem is that I can't get more threads serializing objects >>>>>>>>>>>>>>>>>> to a node. >>>>>>>>>>>>>>>>>> Example: one client I have 30%, lets say. If I add other >>>>>>>>>>>>>>>>>> client talking to >>>>>>>>>>>>>>>>>> other actor instance in parallel I would expect around 60% >>>>>>>>>>>>>>>>>> cpu usage in my >>>>>>>>>>>>>>>>>> server (I have 6 threads minimum and I'm pretty sure that >>>>>>>>>>>>>>>>>> the configuration >>>>>>>>>>>>>>>>>> would enable that from workers and netty perspective). >>>>>>>>>>>>>>>>>> Nevertheless I get >>>>>>>>>>>>>>>>>> around 40% of my 8 cores machine working. If I put the >>>>>>>>>>>>>>>>>> actors in different >>>>>>>>>>>>>>>>>> processes I get the 60% I was expecting. When I say server I >>>>>>>>>>>>>>>>>> mean an >>>>>>>>>>>>>>>>>> actorsystem process (a single java process). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks again for your help. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sun, Jul 3, 2016 at 11:55 AM, Viktor Klang < >>>>>>>>>>>>>>>>>> viktor...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Eduardo, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I meant the overhead of the Java Serialization envelope. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sat, Jul 2, 2016 at 10:12 PM, Eduardo Fernandes < >>>>>>>>>>>>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Viktor. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I'm using basic (binary) serialization of basic Java >>>>>>>>>>>>>>>>>>>> types (int, String (UTF), long, arrays of basic types, >>>>>>>>>>>>>>>>>>>> etc...). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The overhead is that depending of internal values there >>>>>>>>>>>>>>>>>>>> is no send to serialize some members and other not. If you >>>>>>>>>>>>>>>>>>>> merge your >>>>>>>>>>>>>>>>>>>> functional logic with the serialization you can make >>>>>>>>>>>>>>>>>>>> optimizations that a >>>>>>>>>>>>>>>>>>>> generic serializar can't do. Example. Suppose that if a >>>>>>>>>>>>>>>>>>>> member A has a null >>>>>>>>>>>>>>>>>>>> value you don't have to serialize other member B. Maybe >>>>>>>>>>>>>>>>>>>> the member B you >>>>>>>>>>>>>>>>>>>> don't have to serialize could have a null value which is >>>>>>>>>>>>>>>>>>>> fast to serialize >>>>>>>>>>>>>>>>>>>> but it is even faster you don't have even to serialize the >>>>>>>>>>>>>>>>>>>> null. This type >>>>>>>>>>>>>>>>>>>> of overhead is only possible if the serializer knows about >>>>>>>>>>>>>>>>>>>> your functional >>>>>>>>>>>>>>>>>>>> logic. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Sat, Jul 2, 2016 at 10:05 PM, Viktor Klang < >>>>>>>>>>>>>>>>>>>> viktor...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Eduardo, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Perhaps I misunderstood, what serialization format are >>>>>>>>>>>>>>>>>>>>> you emitting in your readObject/writeObject? >>>>>>>>>>>>>>>>>>>>> What overhead are you observing compared to using a >>>>>>>>>>>>>>>>>>>>> custom Serializer? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Sat, Jul 2, 2016 at 10:02 PM, Eduardo Fernandes < >>>>>>>>>>>>>>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> If you have writeObject/readObject defined in your >>>>>>>>>>>>>>>>>>>>>> class the Java plain serialization will invoke those >>>>>>>>>>>>>>>>>>>>>> methods. In my case >>>>>>>>>>>>>>>>>>>>>> all my internal members and class references are also >>>>>>>>>>>>>>>>>>>>>> serialized using the >>>>>>>>>>>>>>>>>>>>>> very same technique. So this is equivalent to >>>>>>>>>>>>>>>>>>>>>> technologies like kryo and >>>>>>>>>>>>>>>>>>>>>> similars since there is no overhead if you serialize >>>>>>>>>>>>>>>>>>>>>> basic members. In >>>>>>>>>>>>>>>>>>>>>> other words the pre-compiles classes you get from kryo >>>>>>>>>>>>>>>>>>>>>> are already made so >>>>>>>>>>>>>>>>>>>>>> there is no performance enhancement in this case. The >>>>>>>>>>>>>>>>>>>>>> big advantage of kryo >>>>>>>>>>>>>>>>>>>>>> is that you don't have to create the >>>>>>>>>>>>>>>>>>>>>> writeObject/readObject by yourself. In >>>>>>>>>>>>>>>>>>>>>> my particular case I've already done that job and my >>>>>>>>>>>>>>>>>>>>>> serialization is >>>>>>>>>>>>>>>>>>>>>> optimized in particular cases where I don't have to >>>>>>>>>>>>>>>>>>>>>> serialize all members >>>>>>>>>>>>>>>>>>>>>> depending of my semantic. I've made some tests and doing >>>>>>>>>>>>>>>>>>>>>> this way is faster >>>>>>>>>>>>>>>>>>>>>> than kryo but you have to burn some calories >>>>>>>>>>>>>>>>>>>>>> implementing a optimized >>>>>>>>>>>>>>>>>>>>>> serialization code. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Bests regards and thanks for your comment. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Sat, Jul 2, 2016 at 9:27 PM, Viktor Klang < >>>>>>>>>>>>>>>>>>>>>> viktor...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I'm not sure I understand why write/readObject >>>>>>>>>>>>>>>>>>>>>>> special methods would necessarily be faster? Most of >>>>>>>>>>>>>>>>>>>>>>> the waste of Java >>>>>>>>>>>>>>>>>>>>>>> Serialization is its envelopes and using class names >>>>>>>>>>>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Sat, Jul 2, 2016 at 1:14 AM, Eduardo Fernandes < >>>>>>>>>>>>>>>>>>>>>>> edu...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I'm using Akka 2.3.13, Java edition. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I'm making some performance tests and in the same >>>>>>>>>>>>>>>>>>>>>>>> machine with 8 cores I see that the serialization >>>>>>>>>>>>>>>>>>>>>>>> process is my >>>>>>>>>>>>>>>>>>>>>>>> bottleneck. I know that because after an increment of >>>>>>>>>>>>>>>>>>>>>>>> actor cpu usage the >>>>>>>>>>>>>>>>>>>>>>>> throughput is exactly the same. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> My actor system talks to 2 other nodes so I see 2 >>>>>>>>>>>>>>>>>>>>>>>> cores dedicated to serialization. Is is possible to >>>>>>>>>>>>>>>>>>>>>>>> increase the number of >>>>>>>>>>>>>>>>>>>>>>>> threads for serialization? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I'm using standard Java serialization but I have my >>>>>>>>>>>>>>>>>>>>>>>> own serialization implementation in my >>>>>>>>>>>>>>>>>>>>>>>> write/readObject methods so I think >>>>>>>>>>>>>>>>>>>>>>>> that switching to kryo or similar will not enhance too >>>>>>>>>>>>>>>>>>>>>>>> much the throughput. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Many thanks for your help. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> /Eduardo >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>>>> You received this message because you are >>>>>>>>>>>>>>>>>>>>>>>> subscribed to the Google Groups "Akka User List" group. >>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving >>>>>>>>>>>>>>>>>>>>>>>> emails from it, send an email to >>>>>>>>>>>>>>>>>>>>>>>> akka-user+...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>>>>>>>>> akka...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>>>>> Visit this group at >>>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user. >>>>>>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>>> √ >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed >>>>>>>>>>>>>>>>>>>>>>> to a topic in the Google Groups "Akka User List" group. >>>>>>>>>>>>>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and all its topics, >>>>>>>>>>>>>>>>>>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>>>>>>>> akka...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>>>> Visit this group at >>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user. >>>>>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed >>>>>>>>>>>>>>>>>>>>>> to the Google Groups "Akka User List" group. >>>>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving >>>>>>>>>>>>>>>>>>>>>> emails from it, send an email to >>>>>>>>>>>>>>>>>>>>>> akka-user+...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>>>>>>> akka...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>>> Visit this group at >>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user. >>>>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>> √ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed >>>>>>>>>>>>>>>>>>>>> to a topic in the Google Groups "Akka User List" group. >>>>>>>>>>>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and all its topics, >>>>>>>>>>>>>>>>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>>>>>> akka...@googlegroups.com. >>>>>>>>>>>>>>>>>>>>> Visit this group at >>>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user. >>>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>>>> the Google Groups "Akka User List" group. >>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving >>>>>>>>>>>>>>>>>>>> emails from it, send an email to >>>>>>>>>>>>>>>>>>>> akka-user+...@googlegroups.com. >>>>>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>>>>> akka...@googlegroups.com. >>>>>>>>>>>>>>>>>>>> Visit this group at >>>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user. >>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>> √ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>>> a topic in the Google Groups "Akka User List" group. >>>>>>>>>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>> To unsubscribe from this group and all its topics, send >>>>>>>>>>>>>>>>>>> an email to akka-user+...@googlegroups.com. >>>>>>>>>>>>>>>>>>> To post to this group, send email to >>>>>>>>>>>>>>>>>>> akka...@googlegroups.com. >>>>>>>>>>>>>>>>>>> Visit this group at >>>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user. >>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>>>>>>>>> >>>>>>>>>> Search the archives: <a href=" >>>>>>>>>>>>>>>>>> https://groups.google.com/group/akka-user"; >>>>>>>>>>>>>>>>>> rel="nofollow" target="_blank" >>>>>>>>>>>>>>>>>> onmousedown="this.href='https:// >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>> in the Google Groups "Akka User List" group. >>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>>>> . >>>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>>> to akka-user+...@googlegroups.com. >>>>>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "Akka User List" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe >>>>>>>>> . >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to akka-user+unsubscr...@googlegroups.com. >>>>>>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>> https://groups.google.com/group/akka-user >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Akka User List" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to akka-user+unsubscr...@googlegroups.com. >>>>>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Patrik Nordwall >>>>>>> Akka Tech Lead >>>>>>> Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM >>>>>>> Twitter: @patriknw >>>>>>> >>>>>>> -- >>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>> >>>>>>>>>> Search the archives: >>>>>>> https://groups.google.com/group/akka-user >>>>>>> --- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "Akka User List" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> akka-user+unsubscr...@googlegroups.com. >>>>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: >>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: >>>>>> https://groups.google.com/group/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to akka-user+unsubscr...@googlegroups.com. >>>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Patrik Nordwall >>>>> Akka Tech Lead >>>>> Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM >>>>> Twitter: @patriknw >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Akka User List" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> akka-user+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+unsubscr...@googlegroups.com. >>>> To post to this group, send email to akka-user@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Akka Tech Lead >>> Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Akka User List" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> akka-user+unsubscr...@googlegroups.com. >>> To post to this group, send email to akka-user@googlegroups.com. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Akka Tech Lead > Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to a topic in the > Google Groups "Akka User List" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/akka-user/EVsIxMEDKeI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
