voonhous commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3850693163
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaEvolutionUtils.scala:
##########
@@ -203,4 +212,125 @@ object ParquetSchemaEvolutionUtils {
internalSchemaOpt
}
}
+
+ /**
+ * Fails fast when schema-on-read meets a shredded variant file. The
internal schema models a
+ * variant as a two-field {metadata, value} record (with sentinel negative
field ids, see
+ * InternalSchemaConverter), so the merged request clips the file's
typed_value away and the
+ * typed rows would read back with a null value residual - silent data loss.
Reconstruction
+ * under schema-on-read is tracked by #18285; until then the read must fail
loudly. The check
+ * anchors on the sentinel ids, which no real user field can carry, so plain
user structs of
+ * the same shape are left alone. The walk recurses through structs, arrays
and maps because
+ * the row writer shreds nested variants too (see VariantSchemaUtils).
+ *
+ * A scan rewritten by Spark's PushVariantIntoScan (4.x) fails fast
regardless of the file's
+ * layout: the merged internal-schema request materializes the variant as
{metadata, value}
+ * while downstream codegen expects the rewrite's ordinal-named extraction
struct, so the
+ * read cannot be served either way (pruning treats the rewritten struct as
the variant
+ * column itself, see SparkInternalSchemaConverter.isVariantRewriteStruct).
+ *
+ * Shared by [[ParquetSchemaEvolutionUtils.getHadoopConfClone]] and the
per-version legacy
+ * file formats, which carry a copy of the same schema-merge block. Callers
gate on a
+ * non-empty projection: empty-projection queries (count(*), select 1) read
no column data
+ * and must keep working, and the query schema is unpruned in that case.
+ */
+ def validateNoShreddedVariants(requiredSchema: StructType, querySchema:
InternalSchema, footerFileMetaData: FileMetaData): Unit = {
+ findVariantRewritePath(requiredSchema).foreach { path =>
+ throw new HoodieException(String.format(
+ "Column '%s' is a variant projected through Spark's variant rewrite "
+ + "(spark.sql.variant.pushVariantIntoScan) and the table is read
with schema-on-read "
+ + "(hoodie.schema.on.read.enable), which cannot reconstruct variants
(see issue "
+ + "#18285). Read without schema-on-read.", path))
+ }
+ val fileParquetSchema = footerFileMetaData.getSchema
+ querySchema.getRecord.fields().foreach { field =>
+ if (fileParquetSchema.containsField(field.name())) {
Review Comment:
Left to #18285. I tried resolving the footer column through the file-named
merged schema and pinning it with a `rename column v to w` leg: the lookup
resolves (query `w#6`, merged `v#6`), but the read still dies in codegen
(`SpecificInternalRow` cast to `VariantVal`) through a path this guard does not
front, so the leg cannot be made green here. Reverted to the query-name lookup
and the javadoc now says a renamed column is not matched here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
