Re: [akka-user] Persistence with Avro
Protobuf, sometimes JSON are the dominating ones I think.
--
Konrad Malawski
On 9 December 2016 at 18:33:42, Richard Rodseth (rrods...@gmail.com) wrote:
> Yes, I've read the docs. Would you say most Lightbend customers use
> Protobuf? I was leaning towards Avro since it seems to be popular in Kafka
> land.
>
> On Fri, Dec 9, 2016 at 6:13 AM, Konrad Malawski <
> konrad.malaw...@typesafe.com> wrote:
>
>> Hi Richard,
>> I fear not that many people use Avro.
>> But I can in general answer your question about SerializerWithStringManif
>> est - yeah it's usually the right way, since it's the most evolvable
>> (you could even remove classes since it's not keyed by classes strictly).
>> So that seems fine.
>>
>> Making sure you've seen: http://doc.akka.io/docs/akka/current/scala/
>> persistence-schema-evolution.html
>>
>> Happy hakking
>>
>> On Wed, Dec 7, 2016 at 12:53 AM, Richard Rodseth
>> wrote:
>>
>>> Anyone else using Avro and avro4s to serialize persistence events? I was
>>> able to get the SerializerWithStringManifest below working (with
>>> generic serialize and deserialize methods). Should I be extending something
>>> other than SerializerWithStringManifest, given that Avro on its own
>>> helps with schema evolution?
>>>
>>> Note: avro4s seems very promising, although I did run into this
>>> limitation
>>> https://github.com/sksamuel/avro4s/issues/75
>>>
>>> class PersistenceEventsAvroSerializer extends
>>> SerializerWithStringManifest {
>>>
>>> val FooManifest = "Foo"
>>>
>>> val BarManifest = "Bar"
>>>
>>> def identifier = 1234567
>>>
>>> override def manifest(obj: AnyRef): String =
>>>
>>> obj match {
>>>
>>> case _: Bar => BarManifest
>>>
>>> case _: Foo => FooManifest
>>>
>>> }
>>>
>>> override def toBinary(obj: AnyRef): Array[Byte] = {
>>>
>>>obj match {
>>>
>>> case foo: Foo => serialize[Foo](foo)
>>>
>>> case bsr: Bar => serialize[Bar](bar)
>>>
>>> }
>>>
>>> }
>>>
>>> override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef
>>> = {
>>>
>>> manifest match {
>>>
>>> case FooManifest => deserialize[Foo](bytes)
>>>
>>> case BarManifest => deserialize[Bar](bytes)
>>>
>>> }
>>>
>>> }
>>>
>>> private def serialize[T](data: T)(implicit s: SchemaFor[T], r:
>>> ToRecord[T]): Array[Byte] = {
>>>
>>> val baos = new ByteArrayOutputStream
>>>
>>> val output = AvroOutputStream.binary[T](baos)
>>>
>>> output.write(data)
>>>
>>> output.close()
>>>
>>> val result = baos.toByteArray()
>>>
>>> result
>>>
>>> }
>>>
>>>
>>> private def deserialize[T](data: Array[Byte])(implicit s:
>>> SchemaFor[T], r: FromRecord[T]): T = {
>>>
>>>
>>> val input = AvroInputStream.binary[T](data)
>>>
>>> val result: T = if (input.iterator.isEmpty) {
>>>
>>> ??? // TODO Check
>>>
>>> } else {
>>>
>>> input.iterator.toSeq.head
>>>
>>> }
>>>
>>> result
>>>
>>> }
>>>
>>>
>>> }
>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>>> urrent/additional/faq.html
>>> >> Search the archives: https://groups.google.com/grou
>>> p/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+unsubscr...@googlegroups.com.
>>> To post to this group, send email to akka-user@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Cheers,
>> Konrad 'ktoso' Malawski
>> Akka
Re: [akka-user] Persistence with Avro
Yes, I've read the docs. Would you say most Lightbend customers use
Protobuf? I was leaning towards Avro since it seems to be popular in Kafka
land.
On Fri, Dec 9, 2016 at 6:13 AM, Konrad Malawski <
konrad.malaw...@typesafe.com> wrote:
> Hi Richard,
> I fear not that many people use Avro.
> But I can in general answer your question about SerializerWithStringManif
> est - yeah it's usually the right way, since it's the most evolvable (you
> could even remove classes since it's not keyed by classes strictly).
> So that seems fine.
>
> Making sure you've seen: http://doc.akka.io/docs/akka/current/scala/
> persistence-schema-evolution.html
>
> Happy hakking
>
> On Wed, Dec 7, 2016 at 12:53 AM, Richard Rodseth
> wrote:
>
>> Anyone else using Avro and avro4s to serialize persistence events? I was
>> able to get the SerializerWithStringManifest below working (with generic
>> serialize and deserialize methods). Should I be extending something other
>> than SerializerWithStringManifest, given that Avro on its own helps with
>> schema evolution?
>>
>> Note: avro4s seems very promising, although I did run into this limitation
>> https://github.com/sksamuel/avro4s/issues/75
>>
>> class PersistenceEventsAvroSerializer extends
>> SerializerWithStringManifest {
>>
>> val FooManifest = "Foo"
>>
>> val BarManifest = "Bar"
>>
>> def identifier = 1234567
>>
>> override def manifest(obj: AnyRef): String =
>>
>> obj match {
>>
>> case _: Bar => BarManifest
>>
>> case _: Foo => FooManifest
>>
>> }
>>
>> override def toBinary(obj: AnyRef): Array[Byte] = {
>>
>>obj match {
>>
>> case foo: Foo => serialize[Foo](foo)
>>
>> case bsr: Bar => serialize[Bar](bar)
>>
>> }
>>
>> }
>>
>> override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef =
>> {
>>
>> manifest match {
>>
>> case FooManifest => deserialize[Foo](bytes)
>>
>> case BarManifest => deserialize[Bar](bytes)
>>
>> }
>>
>> }
>>
>> private def serialize[T](data: T)(implicit s: SchemaFor[T], r:
>> ToRecord[T]): Array[Byte] = {
>>
>> val baos = new ByteArrayOutputStream
>>
>> val output = AvroOutputStream.binary[T](baos)
>>
>> output.write(data)
>>
>> output.close()
>>
>> val result = baos.toByteArray()
>>
>> result
>>
>> }
>>
>>
>> private def deserialize[T](data: Array[Byte])(implicit s:
>> SchemaFor[T], r: FromRecord[T]): T = {
>>
>>
>> val input = AvroInputStream.binary[T](data)
>>
>> val result: T = if (input.iterator.isEmpty) {
>>
>> ??? // TODO Check
>>
>> } else {
>>
>> input.iterator.toSeq.head
>>
>> }
>>
>> result
>>
>> }
>>
>>
>> }
>>
>> --
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>> urrent/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+unsubscr...@googlegroups.com.
>> To post to this group, send email to akka-user@googlegroups.com.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Cheers,
> Konrad 'ktoso' Malawski
> Akka
Re: [akka-user] Persistence with Avro
Hi Richard,
I fear not that many people use Avro.
But I can in general answer your question about SerializerWithStringManifest -
yeah it's usually the right way, since it's the most evolvable (you could
even remove classes since it's not keyed by classes strictly).
So that seems fine.
Making sure you've seen:
http://doc.akka.io/docs/akka/current/scala/persistence-schema-evolution.html
Happy hakking
On Wed, Dec 7, 2016 at 12:53 AM, Richard Rodseth wrote:
> Anyone else using Avro and avro4s to serialize persistence events? I was
> able to get the SerializerWithStringManifest below working (with generic
> serialize and deserialize methods). Should I be extending something other
> than SerializerWithStringManifest, given that Avro on its own helps with
> schema evolution?
>
> Note: avro4s seems very promising, although I did run into this limitation
> https://github.com/sksamuel/avro4s/issues/75
>
> class PersistenceEventsAvroSerializer extends
> SerializerWithStringManifest {
>
> val FooManifest = "Foo"
>
> val BarManifest = "Bar"
>
> def identifier = 1234567
>
> override def manifest(obj: AnyRef): String =
>
> obj match {
>
> case _: Bar => BarManifest
>
> case _: Foo => FooManifest
>
> }
>
> override def toBinary(obj: AnyRef): Array[Byte] = {
>
>obj match {
>
> case foo: Foo => serialize[Foo](foo)
>
> case bsr: Bar => serialize[Bar](bar)
>
> }
>
> }
>
> override def fromBinary(bytes: Array[Byte], manifest: String): AnyRef = {
>
> manifest match {
>
> case FooManifest => deserialize[Foo](bytes)
>
> case BarManifest => deserialize[Bar](bytes)
>
> }
>
> }
>
> private def serialize[T](data: T)(implicit s: SchemaFor[T], r:
> ToRecord[T]): Array[Byte] = {
>
> val baos = new ByteArrayOutputStream
>
> val output = AvroOutputStream.binary[T](baos)
>
> output.write(data)
>
> output.close()
>
> val result = baos.toByteArray()
>
> result
>
> }
>
>
> private def deserialize[T](data: Array[Byte])(implicit s: SchemaFor[T],
> r: FromRecord[T]): T = {
>
>
> val input = AvroInputStream.binary[T](data)
>
> val result: T = if (input.iterator.isEmpty) {
>
> ??? // TODO Check
>
> } else {
>
> input.iterator.toSeq.head
>
> }
>
> result
>
> }
>
>
> }
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
Cheers,
Konrad 'ktoso' Malawski
Akka
