Re: [akka-user] akka-persistence (2.3.4) and serialization
Hi Michael, yes, the TCK is not published yet – it will be as part of the 2.3.5 release. If you would like to use it before that you can publishLocal from the akka-persistence-tck-experimental project from the release-2.3 branch. Good luck with your plugin! On 11 August 2014 at 08:29:22, Michael Reardon (reard...@gmail.com) wrote: I looks like the TCK has not made it up to Maven-land yet, or is that just me? I'm been looking here: http://central.maven.org/maven2/com/typesafe/akka/ Incidentally, I've starting cobbling together a persistence plugin for CouchDB, so I too will be sorting out handling json with Akka Serializer. I've not noticed anyone using CouchDB with Akka Persistence yet, but if someone else has I'd love to hear about it. Thanks, Mike Reardon On Tuesday, August 5, 2014 3:54:56 AM UTC-7, Konrad Malawski wrote: By the way, implementing storage plugins is not so hard :-) Take a look at the docs: http://doc.akka.io/docs/akka/snapshot/scala/persistence.html#storage-plugins (I'm linking to snapshot here because in in 2.3.5 (coming soon) persistence includes an persistence plugin TCK – so you can verify if your plugin really works with this out of the box set of tests :-)) On Tue, Aug 5, 2014 at 12:51 PM, Konrad Malawski kt...@typesafe.com wrote: Hi Yann, since we provide user implementable Journal APIs to which we send PersistentRepr we (as in Akka) don't really have control over the format of how a journal plugin will decide to store this data. This is by design, since we want to be de-coupled from any database specifics in the APIs. In HBase I store a whole lot of data in the row key, an SQL journal would probably utilise separate columns for the persistenceId and the sequenceNr etc. It is up to the journal plugin to decide how the data will be fit into the database (if you're mongo, just slap a json document into the collection and it'll work fine – etc) – which is why my answer here is to implement a Journal plugin that would store the data in a way you see fit – just setting the serialiser won't be enough since the plugin may still do wrapping, and we can't force people to not do so, since perhaps it's exactly what they need. If your case is strong enough to really need full human readability, writing a small plugin shouldn't be to big of a price to pay :-) Related question: which database are you targeting? I hope you see my point and this helps in the long run! -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Konrad 'ktoso' Malawski hAkker @ typesafe http://akka.io -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] akka-persistence (2.3.4) and serialization
Hi Konrad, Thanks for your answer. The development is always about tradeoff, and good tradeoff is always context sensitive. In my current context, performance is not as important as human readability. The following tradeoff is enough for me : - Metadata as binary in one field. - Payload as text in another field. I'm not a big fan of the merge of business data and persistent technical data, but I suppose it's designed for speed. You have your own tradeoff to choose from and I don't discuss them. However, I'm not very pleased by the solution write your own journal plugin, it seems a big task relatively to the original problem which is don't mix your data with mine. I also think that don't mix technical data with business one can be a reasonable requirement. Now that's said, I don't have much idea of a solution which do not impact speed and do not involve journal plugin api break (like separating the PersistentRepr data from the payload to discourage mixing). Le lundi 4 août 2014 12:32:24 UTC+2, Konrad Malawski a écrit : Hello Yann, There is not “nulling” involved. The PersistentRepr acts as envelope for your payload (your message), and adds some metadata to it (persistenceId etc). See details here: https://github.com/akka/akka/blob/master/akka-persistence/src/main/scala/akka/persistence/Persistent.scala#L191 We serialise the entire PersistentRepr and persist it, inside it there will be your payload, serialised with whatever you chose. The goal here is to allow efficient serialisers, as well as to help out in schema evolution (as Endre hinted with protobufs or something similar). If you want to have full control over serialisation, you could implement a persistence-plugin, as then you’re the guy asked to persist the event. In a schema-less db such as mongo you could explode the event json into a document and add fields for persistentId etc… Having that said I’m not a big fan of this aproach, and JSON is not the best format for either speed or schema evolution (bigger reasons than human readability IMO). On Fri, Jul 25, 2014 at 2:00 PM, Yann Le Moigne yann.l...@gmail.com javascript: wrote: Thanks for quick answer. So the journal plugin has to get and serialize the payload of PersistentRepr for readable format, then nullify it in PersistentRepr, then serialize PersistentRepr, and store both serialization result side by side ? The nullify part scary me, probably duplication is better. I have some concerns about durability of core business data which the journal is. I fear that data became unreadable without the help of akka because business data if wrapped in binary technical data. Also for debugging business state, being able to have the journal in a readable format is a major advantage. Le vendredi 25 juillet 2014 13:09:53 UTC+2, Akka Team a écrit : Hi, As PersistentImpl is private, it seems very delicate to write a serializer for it. As the wrapper above is a sensitive internal class you should definitely not try to serialize it in a custom way. If you need finer grained control of your journal format you have to write a journal plugin. So how can I get a clean output ? If I can't, what's the point to allow configuration for custom serializer ? The point is that you have more control about the compatiblity of your event data between versions, for example by carefully constructing your persistence data format in protobuf. Also, java serialization is slow. -Endre Thanks for help. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/ current/additional/faq.html Search the archives: https://groups.google.com/ group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com. To post to this group, send email to akka...@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com javascript:. To post to this group, send email to akka...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com --
Re: [akka-user] akka-persistence (2.3.4) and serialization
Hi Yann, since we provide user implementable Journal APIs to which we send PersistentRepr we (as in Akka) don't really have control over the format of how a journal plugin will decide to store this data. This is by design, since we want to be de-coupled from any database specifics in the APIs. In HBase I store a whole lot of data in the row key, an SQL journal would probably utilise separate columns for the persistenceId and the sequenceNr etc. It is up to the journal plugin to decide how the data will be fit into the database (if you're mongo, just slap a json document into the collection and it'll work fine – etc) – which is why my answer here is to implement a Journal plugin that would store the data in a way you see fit – just setting the serialiser won't be enough since the plugin may still do wrapping, and we can't force people to not do so, since perhaps it's exactly what they need. If your case is strong enough to really need full human readability, writing a small plugin shouldn't be to big of a price to pay :-) Related question: which database are you targeting? I hope you see my point and this helps in the long run! -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] akka-persistence (2.3.4) and serialization
By the way, implementing storage plugins is not so hard :-) Take a look at the docs: http://doc.akka.io/docs/akka/snapshot/scala/persistence.html#storage-plugins (I'm linking to snapshot here because in in 2.3.5 (coming soon) persistence includes an persistence plugin TCK – so you can verify if your plugin really works with this out of the box set of tests :-)) On Tue, Aug 5, 2014 at 12:51 PM, Konrad Malawski kt...@typesafe.com wrote: Hi Yann, since we provide user implementable Journal APIs to which we send PersistentRepr we (as in Akka) don't really have control over the format of how a journal plugin will decide to store this data. This is by design, since we want to be de-coupled from any database specifics in the APIs. In HBase I store a whole lot of data in the row key, an SQL journal would probably utilise separate columns for the persistenceId and the sequenceNr etc. It is up to the journal plugin to decide how the data will be fit into the database (if you're mongo, just slap a json document into the collection and it'll work fine – etc) – which is why my answer here is to implement a Journal plugin that would store the data in a way you see fit – just setting the serialiser won't be enough since the plugin may still do wrapping, and we can't force people to not do so, since perhaps it's exactly what they need. If your case is strong enough to really need full human readability, writing a small plugin shouldn't be to big of a price to pay :-) Related question: which database are you targeting? I hope you see my point and this helps in the long run! -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] akka-persistence (2.3.4) and serialization
I understand your point. :) I think the api encourage to serialize the whole PersistentRepr, as the ddvore mongo plugin does. But I also clearly understand that's a plugin author choice and not yours. And using a plugin mechanism is a great choice, no one want to be coupled to specific database and/or data representation. I'll investigate to write a plugin. And big THANKS for the TCK. Currently I'm prototyping, so the database choice is not definitive, but I go for mongo. (one of the reason is that good GUI tooling is required to allow less technical people to see very quickly what happened in case of bad business state) Yes your answers help. Thank you. Le mardi 5 août 2014 12:54:56 UTC+2, Konrad Malawski a écrit : By the way, implementing storage plugins is not so hard :-) Take a look at the docs: http://doc.akka.io/docs/akka/snapshot/scala/persistence.html#storage-plugins (I'm linking to snapshot here because in in 2.3.5 (coming soon) persistence includes an persistence plugin TCK – so you can verify if your plugin really works with this out of the box set of tests :-)) On Tue, Aug 5, 2014 at 12:51 PM, Konrad Malawski kt...@typesafe.com javascript: wrote: Hi Yann, since we provide user implementable Journal APIs to which we send PersistentRepr we (as in Akka) don't really have control over the format of how a journal plugin will decide to store this data. This is by design, since we want to be de-coupled from any database specifics in the APIs. In HBase I store a whole lot of data in the row key, an SQL journal would probably utilise separate columns for the persistenceId and the sequenceNr etc. It is up to the journal plugin to decide how the data will be fit into the database (if you're mongo, just slap a json document into the collection and it'll work fine – etc) – which is why my answer here is to implement a Journal plugin that would store the data in a way you see fit – just setting the serialiser won't be enough since the plugin may still do wrapping, and we can't force people to not do so, since perhaps it's exactly what they need. If your case is strong enough to really need full human readability, writing a small plugin shouldn't be to big of a price to pay :-) Related question: which database are you targeting? I hope you see my point and this helps in the long run! -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] akka-persistence (2.3.4) and serialization
Hello Yann, There is not “nulling” involved. The PersistentRepr acts as envelope for your payload (your message), and adds some metadata to it (persistenceId etc). See details here: https://github.com/akka/akka/blob/master/akka-persistence/src/main/scala/akka/persistence/Persistent.scala#L191 We serialise the entire PersistentRepr and persist it, inside it there will be your payload, serialised with whatever you chose. The goal here is to allow efficient serialisers, as well as to help out in schema evolution (as Endre hinted with protobufs or something similar). If you want to have full control over serialisation, you could implement a persistence-plugin, as then you’re the guy asked to persist the event. In a schema-less db such as mongo you could explode the event json into a document and add fields for persistentId etc… Having that said I’m not a big fan of this aproach, and JSON is not the best format for either speed or schema evolution (bigger reasons than human readability IMO). On Fri, Jul 25, 2014 at 2:00 PM, Yann Le Moigne yann.lemoi...@gmail.com wrote: Thanks for quick answer. So the journal plugin has to get and serialize the payload of PersistentRepr for readable format, then nullify it in PersistentRepr, then serialize PersistentRepr, and store both serialization result side by side ? The nullify part scary me, probably duplication is better. I have some concerns about durability of core business data which the journal is. I fear that data became unreadable without the help of akka because business data if wrapped in binary technical data. Also for debugging business state, being able to have the journal in a readable format is a major advantage. Le vendredi 25 juillet 2014 13:09:53 UTC+2, Akka Team a écrit : Hi, As PersistentImpl is private, it seems very delicate to write a serializer for it. As the wrapper above is a sensitive internal class you should definitely not try to serialize it in a custom way. If you need finer grained control of your journal format you have to write a journal plugin. So how can I get a clean output ? If I can't, what's the point to allow configuration for custom serializer ? The point is that you have more control about the compatiblity of your event data between versions, for example by carefully constructing your persistence data format in protobuf. Also, java serialization is slow. -Endre Thanks for help. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/ current/additional/faq.html Search the archives: https://groups.google.com/ group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com. To post to this group, send email to akka...@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[akka-user] akka-persistence (2.3.4) and serialization
Hi,
I would like to have readable entry in journal.
To do so, I wrote json serializer for my events and setup up the serializer
in application.conf :
akka {
...
actor {
serializers {
json = my.app.BusinessEventJsonSerializer
}
serialization-bindings {
my.app.MyBusinessEventTrait = json
}
}
}
Now the problem is that my payload seems to be wrapped in an instance of
akka.persistence.PersistentImpl as shown in log :
[debug] a.s.Serialization(akka://application) - Using
serializer[akka.persistence.serialization.MessageSerializer] for message
[akka.persistence.PersistentImpl]
So, when I go in my datastore, I got an ugly string, where my payload is
wrapped in a binary format :
\nU\b�\r\u0012P{\n \type\ : \ServerWatchAdded\,\n \payload\ : {\n
\address\ : \1.1.1.1\\n }\n}\u0010\u0001\u001a\u0006Server
\u0\u@\uZ\u001aakka://application/temp/$a
As PersistentImpl is private, it seems very delicate to write a serializer
for it.
So how can I get a clean output ?
If I can't, what's the point to allow configuration for custom serializer ?
Thanks for help.
--
Read the docs: http://akka.io/docs/
Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups Akka
User List group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] akka-persistence (2.3.4) and serialization
Hi, As PersistentImpl is private, it seems very delicate to write a serializer for it. As the wrapper above is a sensitive internal class you should definitely not try to serialize it in a custom way. If you need finer grained control of your journal format you have to write a journal plugin. So how can I get a clean output ? If I can't, what's the point to allow configuration for custom serializer ? The point is that you have more control about the compatiblity of your event data between versions, for example by carefully constructing your persistence data format in protobuf. Also, java serialization is slow. -Endre Thanks for help. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
