voonhous opened a new issue, #19578: URL: https://github.com/apache/hudi/issues/19578
Follow-up to #19558. Batch incremental queries on current table versions scan through HoodieFileGroupReaderBasedFileFormat with a catalyst-provided schema (covered by a round trip in TestVariantDataType). The paths that build SparkFileFormatInternalRowReaderContext without a catalyst schema and rely on the buildFullVariantReadSchema rewrite are not exercised end to end for variant columns: - Spark structured streaming source (HoodieStreamSourceV2 -> MergeOnReadIncrementalRelationV2 -> HoodieMergeOnReadRDDV2) - CDC queries (CDCFileGroupIterator) - V1 incremental relation on pre-v8 table versions Each of these should get a round trip over a shredded variant table (write shredded, read back through the path, compare values). Base-only splits in HoodieMergeOnReadRDDV2 (dataFileOnlySplit) bypass the reader context entirely and use the relation's base file readers; that leg should be checked for the same null-read symptom while adding coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
