Re: AVRO schema evolution: adding optional column with default fails deserialization

2019-08-01 Thread Svante Karlsson
return result; > } catch (IOException ex) { > throw new SerializationException( > "Can't serialize data='" + data, ex); > } > } > > čt 1. 8. 2019 v 17:06 odesílatel Svante Karlsson > napsal: > >>

Re: AVRO schema evolution: adding optional column with default fails deserialization

2019-08-01 Thread Svante Karlsson
deserializer to enable schema evolution. > > thanks in advance, > Martin > > čt 1. 8. 2019 v 15:55 odesílatel Svante Karlsson > napsal: > >> In an avrofile the schema is in the beginning but if you refer a single >> record serialization like Kafka then you

Re: AVRO schema evolution: adding optional column with default fails deserialization

2019-08-01 Thread Svante Karlsson
In an avrofile the schema is in the beginning but if you refer a single record serialization like Kafka then you have to add something that you can use to get hold of the schema. Confluents avroencoder for Kafka uses confluents schema registry that uses int32 as schema Id. This is prepended (+a

Re: Avro C++: How to serialize data as Data Object File to send in Kafka?

2019-07-12 Thread Svante Karlsson
This is maybe not the nicest implementation since it feels way to complicated but the only on I found out. Checkout encode starting at line 95. Note that the example encodes data using confluent's schema registry format (ie 5 extra bytes) and does a double copy - I have not found a way to get rid

Re: C++ How to get the length of the encoded data

2018-10-27 Thread Svante Karlsson
You need to call flush(). Send common usecaee below auto bin_os = avro::memoryOutputStream(); avro::EncoderPtr bin_encoder = avro::binaryEncoder(); bin_encoder->init(*bin_os.get()); avro::encode(*bin_encoder, src); bin_encoder->flush(); /* push back unused characters to

Re: fromJson is failing with null as uniontype

2016-02-29 Thread Svante Karlsson
The problem is that avro has it's own representation of union encoding so your experience would encode to {"int": 50}. In a recent project we ended up writing a slightly modded json parser to be able to use avro schemas on existing json rest calls. 2016-02-29 9:20 GMT+01:00 Chris Miller

Re: Using Avro for encoding messages

2015-07-09 Thread Svante Karlsson
I had the same problem a while ago and for the same reasons as you mention we decided to use fingerprints (MD5 hash of the schema), however there are some catches here. First I believe that the normalisation of the schema is incomplete so you might end up with different hashes of the same schema.

Re: Using Avro for encoding messages

2015-07-09 Thread Svante Karlsson
What causes the schema normalization to be incomplete? Bad implementation, I use C++ avro and it's not complete and not very active. And is that a problem? As long as the reader can get the schema, it shouldn't matter that there are duplicates – as long as the differences between the duplicates

Re: Schema default values in C++ implementation

2015-04-16 Thread Svante Karlsson
I think you are hit by https://issues.apache.org/jira/browse/AVRO-1335 I recently extended the avrogen_cpp thing so it also generates the following members to your class. ... static inline const boost::uuids::uuid schema_hash() { static const boost::uuids::uuid

Re: Deserialize Avro Object Without Schema

2015-03-25 Thread svante karlsson
The schema is written inside an avro file. Thats why you don't need to provide it. You really need the schema to decode avro data. Either by providing a schema from somewhere and using a generic datum reader or by generating a hardcoded decoder that knows the schema from compile time. regards

Re: Error building/running tests on avro c++

2014-08-01 Thread svante karlsson
I had some issues with the cmakefile when I built avro c++ for windows a month or two ago. If I remebered correctly it did not find or possibly figure out the configuration of boost. I ended up doing some small hacks in the CMakeList.txt file to get it to compile. This was on windows so the

Re: Error building/running tests on avro c++

2014-08-01 Thread svante karlsson
Since your using solaris check the ticket below: (it speaks about a bug in 1.53 that has been fixed in 1.54) https://svn.boost.org/trac/boost/ticket/8212 /svante 2014-08-01 15:28 GMT+02:00 jeff saremi jeffsar...@hotmail.com: Svente, thanks very much for the info. I looked at the shell file

128 bit integers

2014-05-28 Thread svante karlsson
I'm having issues with endian converison of 128 bit integers (uuid's in my case) but the problem is generic I currently encodes them as fixed but that leaves the swapping of bytes (for endianness) up to the user. I had not given the matter any thought until we streched some existing 64 bit id's

c++11 http client and server library

2014-04-26 Thread svante karlsson
I've started to work on a c++ library that I intend to use for performing avro encoded rest calls. If/when I understand how to implement avro rpc it should be simple enough to extend the existing code base to that as well. The client is implemented using libcurl and boost asio. The server is