Hi all,
I do not have a lot of experience using Avro, so hopefully this is not an
obvious question (and I hope this is the right place to ask):
I have a schema with a decimal logical type, e.g.,
{
"type" : "record",
"name" : "testRecord",
"namespace" : "org.example",
"doc" : "",
"fields" : [
{
"name" : "value",
"type":[
"null",
{
"type":"bytes",
"logicalType":"decimal",
"precision":12,
"scale":4
}
],
"doc":"",
"default":null
}
]
}
And I have a code that parses this schema and creates a GenericRecord based
on that then puts a BigDecimal for "value" (I have copied the full code at
the end). The problem is that when I write this record to file, I get the
following exception which IIUC is coming from the fact that there are no
conversions registered for BigDecimal here
<https://github.com/apache/avro/blob/9a6aa43c8699ccde7bef485206928acb35378bdb/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java#L859>
:
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.AvroRuntimeException:
Unknown datum type java.math.BigDecimal: 10.0
at org.apache.avro.file.DataFileWriter.append (DataFileWriter.java:317)
at org.openmrs.analytics.TestAvro.main (TestAvro.java:25)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
at java.lang.Thread.run (Thread.java:834)
Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type
java.math.BigDecimal: 10.0
at org.apache.avro.generic.GenericData.getSchemaName (
GenericData.java:912)
at org.apache.avro.generic.GenericData.resolveUnion (GenericData.java:874
)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion (
GenericDatumWriter.java:272)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion (
GenericDatumWriter.java:143)
at org.apache.avro.generic.GenericDatumWriter.write (
GenericDatumWriter.java:83)
at org.apache.avro.generic.GenericDatumWriter.writeField (
GenericDatumWriter.java:221)
at org.apache.avro.generic.GenericDatumWriter.writeRecord (
GenericDatumWriter.java:210)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion (
GenericDatumWriter.java:131)
at org.apache.avro.generic.GenericDatumWriter.write (
GenericDatumWriter.java:83)
at org.apache.avro.generic.GenericDatumWriter.write (
GenericDatumWriter.java:73)
at org.apache.avro.file.DataFileWriter.append (DataFileWriter.java:314)
at org.openmrs.analytics.TestAvro.main (TestAvro.java:25)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
at java.lang.Thread.run (Thread.java:834)
But my understanding is that BigDecimal is the right type for the Avro
decimal logical type, is that correct? If yes, shouldn't this approach
work? I can do a conversion from BigDecimal to ByteBuffer but that is
something that I want to avoid because in my real use case, I am receiving
GenericRecord from another library that creates the schema too, possibly
with many such logical types. Here is the full code:
import java.io.File;
import java.io.IOException;
import java.math.BigDecimal;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData.Record;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumWriter;
public class TestAvro {
public static void main(String[] args) throws IOException {
Schema testSchema = new Schema.Parser().parse(new
File("tmp/test_decimal_union.avsc"));
GenericRecord testRecord = new Record(testSchema);
testRecord.put("value", BigDecimal.valueOf(10.0));
DatumWriter<GenericRecord> datumWriter = new
GenericDatumWriter<>(testSchema);
DataFileWriter<GenericRecord> dataFileWriter = new
DataFileWriter<GenericRecord>(datumWriter);
File testFile = new File("tmp/test_decimal.avro");
dataFileWriter.create(testSchema, testFile);
dataFileWriter.append(testRecord);
dataFileWriter.close();
}
Any help is appreciated.
-B
