voonhous commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3860853754
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaEvolutionUtils.scala:
##########
@@ -203,4 +212,130 @@ object ParquetSchemaEvolutionUtils {
internalSchemaOpt
}
}
+
+ /**
+ * Fails fast when schema-on-read meets a shredded variant file. The
internal schema models a
+ * variant as a two-field {metadata, value} record (with sentinel negative
field ids, see
+ * InternalSchemaConverter), so the merged request clips the file's
typed_value away and the
+ * typed rows would read back with a null value residual - silent data loss.
Reconstruction
+ * under schema-on-read is tracked by #18285; until then the read must fail
loudly. The check
+ * anchors on the sentinel ids, which no real user field can carry, so plain
user structs of
+ * the same shape are left alone. The walk recurses through structs, arrays
and maps because
+ * the row writer shreds nested variants too (see VariantSchemaUtils).
+ *
+ * Footer columns are resolved by the query-schema name. A column renamed
under schema-on-read
+ * still carries its old name in the file and is not matched here; such
reads are left to
+ * #18285 with reconstruction itself.
+ *
+ * A scan rewritten by Spark's PushVariantIntoScan (4.x) fails fast
regardless of the file's
+ * layout: the merged internal-schema request materializes the variant as
{metadata, value}
+ * while downstream codegen expects the rewrite's ordinal-named extraction
struct, so the
+ * read cannot be served either way (pruning treats the rewritten struct as
the variant
+ * column itself, see SparkInternalSchemaConverter.isVariantRewriteStruct).
+ *
+ * Shared by [[ParquetSchemaEvolutionUtils.getHadoopConfClone]] and the
per-version legacy
+ * file formats, which carry a copy of the same schema-merge block. Callers
gate on a
+ * non-empty projection: empty-projection queries (count(*), select 1) read
no column data
+ * and must keep working, and the query schema is unpruned in that case.
+ */
+ def validateNoShreddedVariants(requiredSchema: StructType, querySchema:
InternalSchema, footerFileMetaData: FileMetaData): Unit = {
+ findVariantRewritePath(requiredSchema).foreach { path =>
Review Comment:
Measured on this branch against a committed internal schema (the
schema-on-read DDL commits one; the insert alone does not) on an unshredded
variant table: inline compaction, inline clustering and CDC reads do reach this
arm and fail with the `pushVariantIntoScan` wording; `run_compaction` /
`run_clustering` pass (their clients carry no internal schema), and so does a
DataFrame upsert through the merge handle (its base-file read never enters this
branch). Skipping Hudi's own shape would not make the services work: the merged
request comes back as `{metadata, value}` where `variantRestoreProjection`
expects the ordinal struct, so it would misread instead of fail; before the
guard the same reads died inside pruning (`cannot prune col: v.0`). Real
support is #18285. Done in ee5c84f17d69: the message names both routes and
points at the procedures, the Javadoc says which reader contexts carry the
internal schema, and a suite leg pins inline compaction (MOR) and inline
clustering (COW)
under schema-on-read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
