[ https://issues.apache.org/jira/browse/AVRO-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100968#comment-17100968 ]
Ryan Skraba commented on AVRO-2817: ----------------------------------- For info: it looks like the schema is invalid: the default value can't be null if the field can't be null. At first glance, this looks like this was tightened up with AVRO-2369. {code} { "type" : "record", "name" : "DemoRecord", "namespace" : "com.mxj142.avro.repo", "fields" : [ { "name" : "enrichmentHeader", "type" : { "type" : "record", "name" : "EnrichmentHeader", "namespace" : "com.mxj142.commons.domain", "fields" : [ { "name" : "correlationId", "type" : { "type" : "string", "avro.java.string" : "String" } }, { "name" : "sourceId", "type" : { "type" : "string", "avro.java.string" : "String" } } ] }, "doc" : "Enrichment header", "default" : null }, { "name" : "cx", "type" : { "type" : "record", "name" : "CXRecord", "fields" : [ { "name" : "dateFirst", "type" : "string" }, { "name" : "dateLast", "type" : "string" } ] } } ] } {code} It's really unfortunate that an error was never thrown on the schema in earlier versions of the Java SDK -- it doesn't match what the spec says. We can currently turn off validating defaults with [{{Schema.Parser.setValidateDefaults}}|https://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.Parser.html#setValidateDefaults-boolean-]. Would it be sufficient to have an option in avrotool to set this to true? > Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for > certian type of schemas > ------------------------------------------------------------------------------------------------------ > > Key: AVRO-2817 > URL: https://issues.apache.org/jira/browse/AVRO-2817 > Project: Apache Avro > Issue Type: Bug > Components: java > Affects Versions: 1.9.2 > Reporter: Manoj Kumar Jha > Priority: Blocker > Fix For: 1.8.2 > > Attachments: records.avro > > > Hello, > We observed avro file generated using avro-1.8.2 is not readable while > using avro-1.9.2 if schema contains a field "default:null". Please see > below. > > -------------------------------------------------------------------------- > [mxj142:Jars]$ ls -ltr > total 168520 > -rwx------@ 1 mxj142 staff 34798932 Jan 16 14:45 avro-tools-1.8.2.jar > -rwxr-xr-x@ 1 mxj142 staff 51303364 Mar 6 17:43 avro-tools-1.9.2.jar > -rw-r--r-- 1 mxj142 staff 715 Apr 24 11:19 records.avro > [mxj142:Jars]$ java -jar avro-tools-1.8.2.jar tojson records.avro > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > {"enrichmentHeader":\{"correlationId":"1","sourceId":"09b5d770-7373-462d-9517-8e5ce957793f"},"cx":\{"dateFirst":"20200419","dateLast":"20200419"}} > {"enrichmentHeader":\{"correlationId":"2","sourceId":"0285dc49-cc5c-4717-8ea2-45e50de2f65f"},"cx":\{"dateFirst":"20200420","dateLast":"20200420"}} > [mxj142:Jars]$ > [mxj142:Jars]$ > [mxj142:Jars]$ > [mxj142:Jars]$ > [mxj142:Jars]$ > [mxj142:Jars]$ java -jar avro-tools-1.9.2.jar tojson records.avro > 20/04/24 11:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Exception in thread "main" org.apache.avro.AvroTypeException: Invalid default > for field enrichmentHeader: null not a > \{"type":"record","name":"EnrichmentHeader","namespace":"com.mxj142.commons.domain","fields":[{"name":"correlationId","type":{"type":"string","avro.java.string":"String"}},\{"name":"sourceId","type":{"type":"string","avro.java.string":"String"}}]} > at org.apache.avro.Schema.validateDefault(Schema.java:1540) > at org.apache.avro.Schema.access$500(Schema.java:87) > at org.apache.avro.Schema$Field.<init>(Schema.java:521) > at org.apache.avro.Schema.parse(Schema.java:1647) > at org.apache.avro.Schema$Parser.parse(Schema.java:1394) > at org.apache.avro.Schema$Parser.parse(Schema.java:1382) > at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:130) > at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:90) > at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:93) > at org.apache.avro.tool.Main.run(Main.java:66) > at org.apache.avro.tool.Main.main(Main.java:55) > [mxj142:Jars]$ > -------------------------------------------------------------------------- > > The file "records.avro" is generated using avro-1.8.2 . It contains two > records and using "avro-tools-1.8.2.jar" one can read it. If we use > "avro-tools-1.9.2.jar", then "records.avro" is not readable (as shown > above). The file "records.avro" is also attached for your convenience. > > Let me know if you need any more information. > > Thanks, > Manoj > > -- This message was sent by Atlassian Jira (v8.3.4#803005)