hudi-agent commented on code in PR #20160:
URL: https://github.com/apache/hudi/pull/20160#discussion_r4145240132
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -102,15 +103,30 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
})
}
+ // Whether the query carries a Spark 4.1 PushVariantIntoScan projection at
all.
+ private lazy val hasVariantProjection: Boolean =
+ sparkRequiredSchema.exists(_.fields.exists(f =>
sparkAdapter.containsVariantProjection(f.dataType)))
+
+ // HoodieFileGroupReader installs the schema handler before it asks for the
output converter, so
+ // this is where the record context learns what shape its rows carry.
Base-file rows are read in
+ // the projected shape and getLogBlockRecordProjection rewrites log rows
into it, so every row
+ // writer the record context builds from an engine schema needs the shape
(#20040).
+ override def setSchemaHandler(schemaHandler:
FileGroupReaderSchemaHandler[InternalRow]): Unit = {
+ super.setSchemaHandler(schemaHandler)
+ if (hasVariantProjection) {
Review Comment:
🤖 I'm not sure this branch projects log records for every merger.
`FileGroupRecordBuffer.getProjectedTransformer` returns before
`getLogBlockRecordProjection` whenever `payloadClasses.isPresent()`, so log
rows on payload-based tables stay `VariantType`. With the shape installed
anyway, `toBinaryRow` on an Avro log row would call `getStruct` on a
`VariantVal`, which throws `ClassCastException`. Could this mirror master's
`hasVariantProjection && (!isPayloadBasedMerge || !getHasLogFiles)` guard?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
