Hudi relies on avro's Schema compatability check. Looks like as per avro SchemaCompatability, INT can't be evolved to a LONG, but LONG to INT is allowed.
Check line no 339 here <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/SchemaCompatibility.java> . Also, check their test case here <https://github.com/apache/avro/blob/master/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibilityTypeMismatch.java> at line 44. On Mon, Sep 7, 2020 at 12:02 PM Prashant Wason <[email protected]> wrote: > Yes, the schema change looks fine. That would mean its an issue with the > schema compatibility checker. The are explicit checks for such cases so > can't say where the issue lies. > > I am out on a vacation this week. I will look into this as soon as I am > back. > > Thanks > Prashant > > On Sun, Sep 6, 2020, 11:18 AM Vinoth Chandar <[email protected]> wrote: > > > That does sound like a backwards compatible change. > > @prashant , any ideas here? (since you have the best context on the > schema > > validation checks) > > > > On Thu, Sep 3, 2020 at 8:12 PM cadl <[email protected]> wrote: > > > > > Hi All, > > > > > > I want to change the type of one column in my COW table, from int to > > long. > > > When I set “hoodie.avro.schema.validate = true” and upsert new data > with > > > long type, I got a “Failed upsert schema compatibility check” error. > > Dose > > > it break backwards compatibility? If I disable > > hoodie.avro.schema.validate, > > > I can upsert and read normally. > > > > > > > > > code demo: > https://gist.github.com/cadl/be433079747aeea88c9c1f45321cc2eb > > > > > > stacktrace: > > > > > > > > > org.apache.hudi.exception.HoodieUpsertException: Failed upsert schema > > > compatibility check. > > > at > > > > > > org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:572) > > > at > > > > > > org.apache.hudi.client.HoodieWriteClient.upsert(HoodieWriteClient.java:190) > > > at > > > > > > org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:260) > > > at > > > > > > org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:169) > > > at > > org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125) > > > at > > > > > > org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) > > > at > > > > > > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) > > > at > > > > > > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) > > > at > > > > > > org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86) > > > at > > > > > > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) > > > at > > > > > > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) > > > at > > > > > > org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) > > > at > > > > > > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > > > at > > > > > > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) > > > at > > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) > > > at > > > > > > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) > > > at > > > > > > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) > > > at > > > > > > org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676) > > > at > > > > > > org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676) > > > at > > > > > > org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) > > > at > > > > > > org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) > > > at > > > > > > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) > > > at > > > > > > org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676) > > > at > > > > > > org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285) > > > at > org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271) > > > at > org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229) > > > ... 69 elided > > > Caused by: org.apache.hudi.exception.HoodieException: Failed schema > > > compatibility check for writerSchema > > > > > > :{"type":"record","name":"foo_record","namespace":"hoodie.foo","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"a","type":"long"},{"name":"b","type":"string"},{"name":"__row_key","type":"int"},{"name":"__row_version","type":"int"}]}, > > > table schema > > > > > > :{"type":"record","name":"foo_record","namespace":"hoodie.foo","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"a","type":"int"},{"name":"b","type":"string"},{"name":"__row_key","type":"int"},{"name":"__row_version","type":"int"}]}, > > > base path :file:///jfs/cadl/hudi_data/schema/foo > > > at > > org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:564) > > > at > > > > > > org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:570) > > > ... 94 more. > > > -- Regards, -Sivabalan
