comphead commented on PR #19674: URL: https://github.com/apache/datafusion/pull/19674#issuecomment-3761419751
Thanks @kosiew I have also tested cast with structs as this PR can help Comet with https://github.com/apache/datafusion-comet/pull/3052 About overlap this is what Spark does when no overlap and field count is different ``` scala> spark.sql(" select struct(1, 2) as str").write.mode("overwrite").parquet("/tmp/test_struct") scala> spark.read.parquet("/tmp/test_struct").printSchema root |-- str: struct (nullable = true) | |-- col1: integer (nullable = true) | |-- col2: integer (nullable = true) scala> spark.read.parquet("/tmp/test_struct").show(false) +------+ |str | +------+ |{1, 2}| +------+ scala> spark.read.schema("str struct<col3 int, col4 int, col5 int>").parquet("/tmp/test_struct").show(false) +----+ |str | +----+ |NULL| +----+ ``` However if there is an overlap ``` scala> spark.read.schema("str struct<col3 int, col4 int, col5 int, col1: int>").parquet("/tmp/test_struct").show(false) +---------------------+ |str | +---------------------+ |{NULL, NULL, NULL, 1}| +---------------------+ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
