[ https://issues.apache.org/jira/browse/AVRO-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224936#comment-17224936 ]
Werner Daehn commented on AVRO-2952: ------------------------------------ Is the travis error: [INFO] Running org.apache.avro.compiler.specific.TestSpecificCompiler 6477 [ERROR] Tests run: 38, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.783 s <<< FAILURE! - in org.apache.avro.compiler.specific.TestSpecificCompiler 6478 [ERROR] org.apache.avro.compiler.specific.TestSpecificCompiler.testConversionInstanceWithDecimalLogicalTypeDisabled Time elapsed: 0.377 s <<< FAILURE! 6479 org.junit.ComparisonFailure: Should use null for decimal if the flag is off expected:<n[ull]> but was:<n[ew org.apache.avro.Conversions.DecimalConversion()]> caused by me or something else? I cannot check for myself because the avro-compiler package does not build with the error that "Idl" cannot be found. I can't find this class either. Can anybody help me here? > Logical Types and Conversions enhancements > ------------------------------------------ > > Key: AVRO-2952 > URL: https://issues.apache.org/jira/browse/AVRO-2952 > Project: Apache Avro > Issue Type: Improvement > Components: java > Affects Versions: 1.10.1 > Reporter: Werner Daehn > Priority: Critical > > *Summary*: > * AvroDatatype Field.getDataType(): returns an object with common data type > related methods. > * Add more trivial LogicalTypes to allow better database integration, e.g. > the LogicalType VARCHAR(10) is a STRING that carries the information that > only ASCII chars are in the payload and up to 10 chars max. > For example, to set an ENUM value for a field f the call > {{testRecord.put(f.name(), > f.getDataType().convertToRawType(myEnum.male.name()));}} > does all the conversion. I considered adding that conversion to the put() > method but did not for fear of side effects. > *Reasoning*: > I am working with Avro (Kafka) for two years now and have implemented > improvements around Logical Types. These I merged into the Avro code with > zero side effects - pure additions. No breaking changes for other Avro users > but a great help for them. > Imagine you connect two databases via Kafka using Avro as the message payload. > # The first problem you will be facing is that RawTypes and LogicalTypes are > handled differently. For LogicalTypes there are conversion functions that > provide metadata (e.g. getConvertedType returns that a Java Instant is the > best data type for a timestamp-millis plus conversion logic. For raw types > there is no such thing. A Boolean can be provided as true, "TRUE", 1,... > # Second problem will be the lack of getObject()/setObject() methods similar > to JDBC. The result are endless switch-case lists to call the correct > methods. In every single project for every user. > # Number three is the usage of the Converters as such. The intended usage is > to add converters to the GenericData and the reader/writer uses the best > suited converter. What I have seen most people do however is to use the > converters manually and assign the raw value directly. While adding > converters is possible still, the conversion at GenericRecord.put() and > GenericRecord.get() is easy now. > # For a data exchange format like Avro, it is important to carry as much > metadata as possible. For example, purely seen from Avro a STRING data type > is just fine. 99% of the string data types in a database are VARCHAR(length) > and NVARCHAR(length). While putting an ASCII String of length 10 into a > STRING is no problem, on the consumer side the only matching data type is a > NCLOB - the worst for a data base. The LogicalTypes provide such nice methods > to carry such metadata, e.g. a LogicalType VARCHAR(10) backed by a String. > These Logical Types do not have any conversion, they just exist for the > metadata. You have such a thing already with the UUID LogicalType. > > *Changes*: > * A new package logicaltypes created. It includes all new LogicalTypes and > the AvroDataType implementations for the various raw data types. > * The existing LogicalTypes are unchanged. The corresponding classes in the > logicaltype package just extend them. > * For that some LogicalType fields needed to be made public. > * The LogicalTypes return the more detailed logicaltype.* classes. > * A test class created. > -- This message was sent by Atlassian Jira (v8.3.4#803005)