[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356433990 ## File path: python/pyspark/sql/avro/functions.py ## @@ -30,9 +30,10 @@ @since(3.0) def from_avro(data, jsonFormatSchema, options={}): """ -Converts a binary column of avro format into its corresponding catalyst value. The specified -schema must match the read data, otherwise the behavior is undefined: it may fail or return -arbitrary result. +Converts a binary column of avro format into its corresponding catalyst value. If a writer's Review comment: Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356433702 ## File path: external/avro/src/main/scala/org/apache/spark/sql/avro/functions.scala ## @@ -45,9 +45,10 @@ object functions { } /** - * Converts a binary column of avro format into its corresponding catalyst value. The specified - * schema must match the read data, otherwise the behavior is undefined: it may fail or return - * arbitrary result. + * Converts a binary column of avro format into its corresponding catalyst value. If a writer's + * schema is provided in the options, a different (but compatible) schema can be used for reading. Review comment: Good point, thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356249061 ## File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala ## @@ -153,4 +153,79 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession { assert(df.collect().map(_.get(0)) === Seq(Row("one"), Row("two"), Row("three"), Row("four"))) } } + + test("SPARK-27506: roundtrip in to_avro and from_avro with different compatible schemas") { +val df = spark.range(10).select( + struct('id.as("col1"), 'id.cast("string").as("col2")).as("struct") +) +val avroStructDF = df.select(functions.to_avro('struct).as("avro")) +val actualAvroSchema = s""" + |{ + | "type": "record", + | "name": "struct", + | "fields": [ + |{"name": "col1", "type": "int"}, + |{"name": "col2", "type": "string"} + | ] + |} Review comment: I've updated the PR, feel free to comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356248649 ## File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala ## @@ -153,4 +153,79 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession { assert(df.collect().map(_.get(0)) === Seq(Row("one"), Row("two"), Row("three"), Row("four"))) } } + + test("SPARK-27506: roundtrip in to_avro and from_avro with different compatible schemas") { +val df = spark.range(10).select( + struct('id.as("col1"), 'id.cast("string").as("col2")).as("struct") +) +val avroStructDF = df.select(functions.to_avro('struct).as("avro")) +val actualAvroSchema = s""" + |{ + | "type": "record", + | "name": "struct", + | "fields": [ + |{"name": "col1", "type": "int"}, + |{"name": "col2", "type": "string"} + | ] + |} Review comment: I think it should be as: ``` val msg = s""" |{ | | |} |""".stripMargin ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356247604 ## File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala ## @@ -51,9 +51,13 @@ case class AvroDataToCatalyst( override def nullable: Boolean = true + private lazy val avroOptions = AvroOptions(options) + @transient private lazy val avroSchema = new Schema.Parser().parse(jsonFormatSchema) - @transient private lazy val reader = new GenericDatumReader[Any](avroSchema) + @transient private lazy val reader = avroOptions.actualSchema +.map(writer => new GenericDatumReader[Any](new Schema.Parser().parse(writer), avroSchema)) Review comment: actualSchema* and updated :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356200877 ## File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala ## @@ -153,4 +153,79 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession { assert(df.collect().map(_.get(0)) === Seq(Row("one"), Row("two"), Row("three"), Row("four"))) } } + + test("SPARK-27506: roundtrip in to_avro and from_avro with different compatible schemas") { +val df = spark.range(10).select( + struct('id.as("col1"), 'id.cast("string").as("col2")).as("struct") +) +val avroStructDF = df.select(functions.to_avro('struct).as("avro")) +val actualAvroSchema = s""" + |{ + | "type": "record", + | "name": "struct", + | "fields": [ + |{"name": "col1", "type": "int"}, + |{"name": "col2", "type": "string"} + | ] + |} Review comment: I've also looked through the codebase of Spark, but it is rather incoherent. Please let me know if you want to change it, and I'll update it right away. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas
Fokko commented on a change in pull request #26780: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/26780#discussion_r356199894 ## File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala ## @@ -153,4 +153,79 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession { assert(df.collect().map(_.get(0)) === Seq(Row("one"), Row("two"), Row("three"), Row("four"))) } } + + test("SPARK-27506: roundtrip in to_avro and from_avro with different compatible schemas") { +val df = spark.range(10).select( + struct('id.as("col1"), 'id.cast("string").as("col2")).as("struct") +) +val avroStructDF = df.select(functions.to_avro('struct).as("avro")) +val actualAvroSchema = s""" + |{ + | "type": "record", + | "name": "struct", + | "fields": [ + |{"name": "col1", "type": "int"}, + |{"name": "col2", "type": "string"} + | ] + |} Review comment: I've fixed this, but something looks off with Github This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org