[ https://issues.apache.org/jira/browse/SPARK-26709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751974#comment-16751974 ]
Apache Spark commented on SPARK-26709: -------------------------------------- User 'gengliangwang' has created a pull request for this issue: https://github.com/apache/spark/pull/23648 > OptimizeMetadataOnlyQuery does not correctly handle the files with zero record > ------------------------------------------------------------------------------ > > Key: SPARK-26709 > URL: https://issues.apache.org/jira/browse/SPARK-26709 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.1.3, 2.2.3, 2.3.2, 2.4.0 > Reporter: Xiao Li > Assignee: Gengliang Wang > Priority: Blocker > Labels: correctness > > {code:java} > import org.apache.spark.sql.functions.lit > withSQLConf(SQLConf.OPTIMIZER_METADATA_ONLY.key -> "true") { > withTempPath { path => > val tabLocation = path.getAbsolutePath > val partLocation = new Path(path.getAbsolutePath, "partCol1=3") > val df = spark.emptyDataFrame.select(lit(1).as("col1")) > df.write.parquet(partLocation.toString) > val readDF = spark.read.parquet(tabLocation) > checkAnswer(readDF.selectExpr("max(partCol1)"), Row(null)) > checkAnswer(readDF.selectExpr("max(col1)"), Row(null)) > } > } > {code} > OptimizeMetadataOnlyQuery has a correctness bug to handle the file with the > empty records for partitioned tables. The above test will fail in 2.4, which > can generate an empty file, but the underlying issue in the read path still > exists in 2.3, 2.2 and 2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org