voonhous commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3844166532
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java:
##########
@@ -233,4 +247,82 @@ private RecordReader<NullWritable, ArrayWritable>
createBootstrappingRecordReade
true);
}
}
-}
\ No newline at end of file
+
+ /**
+ * The file-group-reader path fails fast on shredded variant reads inside
+ * HiveHoodieReaderContext, but a split can bypass it three ways (see
+ * HoodieInputFormatUtils.shouldUseFilegroupReader): the file group reader
disabled,
+ * schema-on-read enabled, and bootstrap splits. Those land on Hive's plain
parquet reader at
+ * the synced {metadata, value} projection, which silently nulls typed_value
- so repeat the
+ * fail-fast for them. Only reads that request a column holding a shredded
variant fail;
+ * count(*) and projections that skip the variant keep working. The footer
read is gated on a
+ * requested column whose synced Hive type embeds the variant {metadata,
value} shape, so
+ * non-variant tables never pay it; when it does run it mirrors the per-file
readSchema the
+ * file-group-reader path already performs.
+ */
+ private static void validateNoShreddedVariantRead(InputSplit split, JobConf
job) {
+ if (!(split instanceof FileSplit)) {
+ return;
+ }
+ Path filePath = ((FileSplit) split).getPath();
+ if
(!filePath.getName().endsWith(HoodieFileFormat.PARQUET.getFileExtension())) {
+ return;
+ }
+ Set<String> requestedColumns =
Arrays.stream(HoodieColumnProjectionUtils.getReadColumnNames(job))
+ .map(name -> name.toLowerCase(Locale.ROOT))
+ .collect(Collectors.toSet());
+ if (requestedColumns.isEmpty()) {
+ // count(*)-style read: no column data is materialized
+ return;
+ }
+ List<String> ioColumns = HoodieColumnProjectionUtils.getIOColumns(job);
+ List<String> ioColumnTypes =
HoodieColumnProjectionUtils.getIOColumnTypes(job);
+ if (ioColumns.size() != ioColumnTypes.size()) {
+ // The guard is best-effort: a malformed columns/columns.types pairing
must not fail
+ // reads the plain parquet reader would otherwise serve.
+ return;
+ }
+ boolean requestsVariantShapedColumn = false;
+ for (int i = 0; i < ioColumns.size(); i++) {
+ if
(requestedColumns.contains(ioColumns.get(i).toLowerCase(Locale.ROOT))) {
+ String type = ioColumnTypes.get(i).toLowerCase(Locale.ROOT);
+ if (type.contains("metadata:binary") && type.contains("value:binary"))
{
+ requestsVariantShapedColumn = true;
+ break;
+ }
+ }
+ }
+ if (!requestsVariantShapedColumn) {
+ return;
+ }
+ StoragePath storagePath = convertToStoragePath(filePath);
+ HoodieStorage storage = HoodieStorageUtils.getStorage(storagePath,
HadoopFSUtils.getStorageConf(job));
+ HoodieSchema fileSchema =
HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(storagePath).readSchema(storage,
storagePath);
+ if (fileSchema.getType() != HoodieSchemaType.RECORD) {
+ return;
+ }
+ HoodieSchema strippedSchema =
VariantSchemaUtils.stripVariantShredding(fileSchema);
Review Comment:
Confirmed: `readSchema` converts the footer through
`AvroSchemaConverterWithTimestampNTZ`, which yields no VARIANT node, so the
strip was an identity and the guard never fired. Switched to shape detection:
`VariantSchemaUtils.containsShreddedVariantShape` walks the footer-derived
column (records, arrays, maps), anchored per requested column on the
variant-shaped Hive type. Pinned in `TestHoodieParquetInputFormat` through
`getRecordReader` with the file group reader disabled (top-level and nested
shredded files fail naming the column; the unshredded twin and a projection
without the variant keep reading); the new test fails on the previous detection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
