hudi-agent commented on code in PR #18923:
URL: https://github.com/apache/hudi/pull/18923#discussion_r3854476418
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -96,29 +106,46 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
})
}
- // Aligns log-block records with the PushVariantIntoScan-projected variant
shape before
- // they reach the merger. Preserves merger metadata cols (_hoodie_record_key,
- // _tmp_metadata_row_index) which the merger reads by ordinal — projecting
down to the
- // bare required schema would drop them and the merger would read garbage
offsets.
- override def getLogBlockRecordProjection(
- dataBlockSchema: HoodieSchema): HOption[JFunction[InternalRow,
InternalRow]] = {
- val needsProjection = sparkRequiredSchema.exists(_.fields.exists(f =>
f.dataType match {
+ // True only when there is a Spark 4.1 PushVariantIntoScan projection to
apply AND the table is
+ // not using a custom (payload-based) merger. Payload-based tables
round-trip records through
+ // PayloadUpdateProcessor.convertToAvroRecord against a schema that still
types variant fields as
Review Comment:
🤖 nit: in Scala, a parameterless `def` (no `()`) signals a pure,
referentially-transparent property by convention. Since this calls
`getRecordMerger()` — which returns `null` before `setRecordMerger()` runs — it
has observable state-dependence; could you add empty parens (`def
shouldProjectVariants(): Boolean`) to signal callers that this isn't a simple
accessor?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
