*Finally I got the resolution :) *
There is bug in latest kryo version. if we use any collection or Map in
message then it throw exception.
https://github.com/EsotericSoftware/kryo/issues/216
There is a work around mentioned in the above link if you are manually
serializing but if you doing through akka configuration then I don't know
how to resolve with latest version but its working with old version
Below are the version I am using for AKKA and KRYO :-
Akka version 2.3.4
akka-kryo-serialization_2.10-0.3.0.jar
kryo-2.22.jar
and one more thing I wold like to mention that for kryo we don't need
message class to implement serializable (this I never found in any forum,
may be I am naive)
below is the application.conf I have used. I hope this will definitely
help.. :)
akka {
event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "DEBUG"
stdout-loglevel = "DEBUG"
extensions =
["com.romix.akka.serialization.kryo.KryoSerializationExtension$"]
actor {
provider = "akka.cluster.ClusterActorRefProvider"
serializers {
kryo = "com.romix.akka.serialization.kryo.KryoSerializer"
}
serialization-bindings {
"java.util.HashMap" = kryo
"com.xyz.message.IMessage" = kryo
"com.xyz.message.TestObject" = kryo
}
kryo {
type = "nograph"
idstrategy = "incremental"
serializer-pool-size = 20
buffer-size = 92427360
max-buffer-size = -1
use-manifests = false
compression = off
implicit-registration-logging = true
kryo-trace = false
#kryo-custom-serializer-init = "CustomKryoSerializerInitFQCN"
mappings {
"java.util.HashMap" = 26
"com.xyz.message.RecordNewMessage" =25
"com.xyz.message.TestObject"= 27
}
kryo-reference-map = false
classes = [
# fully.qualified.classname1
# fully.qualified.classname2
]
}
}
remote {
#trusted-selection-paths = ["/user/filetransformernode"]
log-remote-lifecycle-events = on
netty.tcp {
hostname = "127.0.0.1"
port = 0
maximum-frame-size = 30809120
#52428800
send-buffer-size = 92427360
receive-buffer-size = 92427360
}
}
cluster {
seed-nodes = [
"akka.tcp://ClusterSystem@127.0.0.1:2551",
"akka.tcp://ClusterSystem@127.0.0.1:2552"]
auto-down-unreachable-after = 300s
}
}
On Monday, November 24, 2014 1:45:41 PM UTC+5:30, neeraj negi wrote:
>
> Yes now I am shifted to use kryo but its not working. below is my
> applicatin .conf
> kindly help because m struck on kryo :(
>
> akka {
> extensions = ["com.romix.akka.serialization.kryo.KryoSerializationExtension$"]
> event-handlers =
> ["akka.event.slf4j.Slf4jEventHandler","com.typesafe.atmos.trace.Slf4jTraceContextEventHandler"]
>
> loggers = ["akka.event.slf4j.Slf4jLogger"]
> loglevel = "ERROR"
>
> actor {
> provider = "akka.cluster.ClusterActorRefProvider"
> serializers {
> # Define kryo serializer
> kryo =
> "com.romix.akka.serialization.kryo.KryoSerializer"
> }
>
> serialization-bindings {
> "java.util.List" = kryo
> "java.io.Serializable" = kryo
> "java.util.Map" = kryo
> "akka.remote.message.Message" = kryo
> }
>
> kryo {
> # Possibles values for type are: graph or nograph
> # graph supports serialization of object graphs with shared nodes
> # and cyclic references, but this comes at the expense of a small
> overhead
> # nograph does not support object grpahs with shared nodes, but is
> usually faster
> type = "graph"
>
>
> # Possible values for idstrategy are:
> # default, explicit, incremental
> #
> # default - slowest and produces bigger serialized representation.
> Contains fully-
> # qualified class names (FQCNs) for each class. Note that selecting this
> strategy
> # does not work in version 0.3.2, but is available on master and from
> 0.3.3 onward.
> #
> # explicit - fast and produces compact serialized representation.
> Requires that all
> # classes that will be serialized are pre-registered using the "mappings"
> and "classes"
> # sections. To guarantee that both sender and receiver use the same
> numeric ids for the same
> # classes it is advised to provide exactly the same entries in the
> "mappings" section
> #
> # incremental - fast and produces compact serialized representation.
> Support optional
> # pre-registering of classes using the "mappings" and "classes" sections.
> If class is
> # not pre-registered, it will be registered dynamically by picking a next
> available id
> # To guarantee that both sender and receiver use the same numeric ids for
> the same
> # classes it is advised to pre-register them using at least the "classes"
> section
>
> idstrategy = "incremental"
>
> # Define a default size for serializer pool
> # Try to define the size to be at least as big as the max possible number
> # of threads that may be used for serialization, i.e. max number
> # of threads allowed for the scheduler
> serializer-pool-size = 16
>
> # Define a default size for byte buffers used during serialization
> buffer-size = 4096
>
> # The serialization byte buffers are doubled as needed until they exceed
> max-buffer-size and an exception is thrown. Can be -1 for no maximum.
> max-buffer-size = -1
>
> # If set, akka uses manifests to put a class name
> # of the top-level object into each message
> use-manifests = false
>
> # Enable transparent compression of serialized messages
> # accepted values are: off | lz4 | deflate
> compression = off
>
> # Log implicitly registered classes. Useful, if you want to know all
> classes
> # which are serialized. You can then use this information in the mappings
> and/or
> # classes sections
> implicit-registration-logging = false
>
> # If enabled, Kryo logs a lot of information about serialization process.
> # Useful for debugging and lowl-level tweaking
> kryo-trace = false
>
> # If proviced, Kryo uses the class specified by a fully qualified class
> name
> # to perform a custom initialization of Kryo instances in addition to what
> # is done automatically based on the config file.
> #kryo-custom-serializer-init = "CustomKryoSerializerInitFQCN"
>
> # Define mappings from a fully qualified class name to a numeric id.
> # Smaller ids lead to smaller sizes of serialized representations.
> #
> # This section is mandatory for idstartegy=explciit
> # This section is optional for idstartegy=incremental
> # This section is ignored for idstartegy=default
> #
> # The smallest possible id should start at 20 (or even higher), because
> # ids below it are used by Kryo internally e.g. for built-in Java and
> # Scala types
> mappings {
> # "akka.remote.message.Message" = 20
> }
>
>
> # If enabled, Kryo uses internally a map detecting shared nodes.
> # This is a preferred mode for big object graphs with a
> lot of nodes.
> # For small object graphs (e.g. below 10 nodes) set it
> to false for
> # better performance.
> kryo-reference-map = true
>
>
> # Define a set of fully qualified class names for
> # classes to be used for serialization.
> # The ids for those classes will be assigned
> automatically,
> # but respecting the order of declaration in this
> section
> #
> # This section is optional for idstartegy=incremental
> # This section is ignored for idstartegy=default
> # This section is optional for idstartegy=explicit
> classes = [
> # fully.qualified.classname1
> # fully.qualified.classname2
> ]
>
> }
>
> }
> remote {
> log-remote-lifecycle-events = off
> netty.tcp {
> hostname = "127.0.0.1"
> port = 0
> maximum-frame-size = 52428800
> send-buffer-size = 52428800
> receive-buffer-size = 52428800
> }
> }
>
> cluster {
> seed-nodes = [
> "akka.tcp://ClusterSystem@127.0.0.1:2551",
> "akka.tcp://ClusterSystem@127.0.0.1:2552"]
>
> auto-down-unreachable-after = 10s
> }
>
>
>
>
>
>
> On Saturday, November 22, 2014 3:20:54 PM UTC+5:30, Akka Team wrote:
>>
>> Hi there,
>> firstly - remoting does not "like" such large messages, it's effectively
>> "cloging the pipe".
>> You should rather try to send more smaller messages to keep the system
>> responsive - sending a large message will take more time to serially
>> serialize it etc.
>>
>> Concerning the speed... You're using the default java serialization -
>> it's slow.
>> If you care about performance - as it looks you do - you should serialize
>> the messages using Kryo, ProtoBuf or CapNproto etc.
>> See here http://doc.akka.io/docs/akka/snapshot/scala/serialization.html
>>
>> On Thu, Nov 20, 2014 at 8:39 AM, neeraj negi <negi.ne...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have two node system one running on 2551 port and other running on
>>> 2552 port on single machine right now.
>>>
>>> I am sending object containing 10000 records from 2551 to 2552 and the
>>> payload size is less then 1 Mb and it's taking around 60 to 70 millisecond
>>> for transfer from one node to another
>>>
>>> and if I increase the records i.e. 70000 and payload becomes around 6 Mb
>>> then it start taking 300 to 400 ms for transfer from one node to another
>>>
>>>
>>> my application conf is:-
>>>
>>> akka {
>>>
>>> event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
>>> loggers = ["akka.event.slf4j.Slf4jLogger"]
>>> loglevel = "debug"
>>>
>>> actor {
>>> provider = "akka.cluster.ClusterActorRefProvider"
>>> }
>>> remote {
>>> log-remote-lifecycle-events = off
>>> netty.tcp {
>>> hostname = "127.0.0.1"
>>> port = 0
>>> maximum-frame-size = 52428800
>>> send-buffer-size = 52428800
>>> receive-buffer-size = 52428800
>>> }
>>> }
>>>
>>> cluster {
>>> seed-nodes = [
>>> "akka.tcp://ClusterSystem@127.0.0.1:2551",
>>> "akka.tcp://ClusterSystem@127.0.0.1:2552"]
>>>
>>> auto-down-unreachable-after = 300s
>>> }
>>>
>>> }
>>>
>>> How can i increase my transfer speed?
>>> am i missing something in Application.conf?
>>>
>>>
>>>
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Akka Team
>> Typesafe - The software stack for applications that scale
>> Blog: letitcrash.com
>> Twitter: @akkateam
>>
>
On Tuesday, November 25, 2014 12:42:32 PM UTC+5:30, neeraj negi wrote:
>
> Hi,
>
> I have tried with `default` id generation strategy
> Below are the logs that i am getting. It's a null pointer exception but I
> am sending only hashmap and a long variable inside the RecordNewMessage
>
>
> 2014-11-25 12:37:56 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] - Metrics will be retreived
> from MBeans, and may be incorrect on some platforms. To increase metric
> accuracy add the 'sigar.jar' to the classpath and the appropriate
> platform-specific native libary to 'java.library.path'. Reason:
> java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
> 2014-11-25 12:37:56 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] - Metrics collection has
> started successfully
> 2014-11-25 12:37:56 INFO LocalActorRef:74 - Message
> [akka.dispatch.sysmsg.Terminate] from
> Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#-864068775]
>
> to
> Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#-864068775]
>
> was not delivered. [1] dead letters encountered. This logging can be turned
> off or adjusted with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 2014-11-25 12:37:56 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://
> ClusterSystem@127.0.0.1:2551] is JOINING, roles [fileconsumernode]
> 2014-11-25 12:37:57 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] to [Up]
> 2014-11-25 12:38:01 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://
> ClusterSystem@127.0.0.1:2552] is JOINING, roles [filetransformernode]
> 2014-11-25 12:38:01 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2552] - Welcome from [akka.tcp://
> ClusterSystem@127.0.0.1:2551]
> transformer node Added
> 2014-11-25 12:38:02 INFO Cluster(akka://ClusterSystem):74 - Cluster Node
> [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node
> [akka.tcp://ClusterSystem@127.0.0.1:2552] to [Up]
> actor selection string is akka.tcp://
> ClusterSystem@127.0.0.1:2552/user/filetransformernode
> process Message
> actor selection string is akka.tcp://
> ClusterSystem@127.0.0.1:2552/user/filetransformernode
> 2014-11-25 12:38:04 ERROR EndpointWriter:65 - AssociationError [akka.tcp://
> ClusterSystem@127.0.0.1:2552] <- [akka.tcp://ClusterSystem@127.0.0.1:2551]:
> Error [java.lang.NullPointerException
> Serialization trace:
> header (com.xyz.message.RecordNewMessage)] [
> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> Serialization trace:
> header (com.xyz.message.RecordNewMessage)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at
> com.romix.akka.serialization.kryo.KryoBasedSerializer.fromBinary(KryoSerializer.scala:394)
> at
> com.romix.akka.serialization.kryo.KryoSerializer.fromBinary(KryoSerializer.scala:239)
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
> at scala.util.Try$.apply(Try.scala:161)
> at akka.serialization.Serialization.deserialize(Serialization.scala:98)
> at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
> at scala.util.Try$.apply(Try.scala:161)
> at akka.serialization.Serialization.deserialize(Serialization.scala:98)
> at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
> at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
> at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
> at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
> at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
> Caused by: java.lang.NullPointerException
> at java.util.HashMap.put(HashMap.java:497)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:144)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> ... 27 more
> ]
> 2014-11-25 12:38:04 ERROR Remoting:66 - java.lang.NullPointerException
> Serialization trace:
> header (com.xyz.message.RecordNewMessage)
> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> Serialization trace:
>
>
>
>
>
>
>
> RecordNewMessage.java class is :-
>
> public class RecordNewMessage implements Serializable{
>
> private static final long serialVersionUID = 1L;
> private final Map<String,String> header;
> private final long startTime;
>
> public RecordNewMessage(Map<String, String> header, long startTime) {
> this.header = header;
> this.startTime=startTime;
> }
> public Map<String, String> getHeader() {
> return header;
> }
>
> public long getStartTime() {
> return startTime;
> }
> }
>
> Can you please let me know where i am wrong with kryo??
>
>
>
> On Saturday, November 22, 2014 3:24:11 PM UTC+5:30, Akka Team wrote:
>>
>> Are both nodes configured to use kryo?
>> What's the error you're getting?
>> Try `default` id generation strategy, like mentioned in another thread
>> this week.
>>
>> --konrad
>>
>> On Thu, Nov 20, 2014 at 7:06 PM, neeraj negi <negi.ne...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have two node system which was working fine till i have added the kryo.
>>>
>>> Attached file is the application.conf that i am using.
>>>
>>> KRYO causing the other node unregistered.
>>>
>>> do i need to change something in the configuration????
>>>
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Akka Team
>> Typesafe - The software stack for applications that scale
>> Blog: letitcrash.com
>> Twitter: @akkateam
>>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
