voonhous commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3850690769
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java:
##########
@@ -233,4 +249,77 @@ private RecordReader<NullWritable, ArrayWritable>
createBootstrappingRecordReade
true);
}
}
-}
\ No newline at end of file
+
+ /**
+ * The file-group-reader path fails fast on shredded variant reads inside
+ * HiveHoodieReaderContext, but a split can bypass it three ways (see
+ * HoodieInputFormatUtils.shouldUseFilegroupReader): the file group reader
disabled,
+ * schema-on-read enabled, and bootstrap splits. Those land on Hive's plain
parquet reader at
+ * the synced {metadata, value} projection, which silently nulls typed_value
- so repeat the
+ * fail-fast for them. Only reads that request a column holding a shredded
variant fail;
+ * count(*) and projections that skip the variant keep working. The footer
read is gated on a
+ * requested column whose synced Hive type embeds the variant {metadata,
value} shape, so
+ * non-variant tables never pay it; when it does run it mirrors the per-file
readSchema the
+ * file-group-reader path already performs. That footer-derived schema
carries no variant
+ * logical type (the converter turns variant groups into plain records), so
the file side is
+ * matched by shape, anchored on the Hive type of the requested column.
+ */
+ @VisibleForTesting
+ static void validateNoShreddedVariantRead(InputSplit split, JobConf job) {
+ if (!(split instanceof FileSplit)) {
+ return;
+ }
+ Path filePath = ((FileSplit) split).getPath();
+ if
(!filePath.getName().endsWith(HoodieFileFormat.PARQUET.getFileExtension())) {
+ return;
+ }
+ Set<String> requestedColumns =
Arrays.stream(HoodieColumnProjectionUtils.getReadColumnNames(job))
+ .map(name -> name.toLowerCase(Locale.ROOT))
+ .collect(Collectors.toSet());
+ if (requestedColumns.isEmpty()) {
+ // count(*)-style read: no column data is materialized
+ return;
+ }
+ List<String> ioColumns = HoodieColumnProjectionUtils.getIOColumns(job);
+ List<String> ioColumnTypes =
HoodieColumnProjectionUtils.getIOColumnTypes(job);
+ if (ioColumns.size() != ioColumnTypes.size()) {
+ // The guard is best-effort: a malformed columns/columns.types pairing
must not fail
+ // reads the plain parquet reader would otherwise serve.
+ return;
+ }
+ // The requested columns whose synced Hive type embeds the variant
{metadata, value} shape:
+ // the anchor for the shape match on the file side below.
+ Set<String> variantColumns = new HashSet<>();
+ for (int i = 0; i < ioColumns.size(); i++) {
+ String name = ioColumns.get(i).toLowerCase(Locale.ROOT);
+ String type = ioColumnTypes.get(i).toLowerCase(Locale.ROOT);
+ if (requestedColumns.contains(name) && type.contains("metadata:binary")
&& type.contains("value:binary")) {
+ variantColumns.add(name);
+ }
+ }
+ if (variantColumns.isEmpty()) {
+ return;
+ }
+ StoragePath storagePath = convertToStoragePath(filePath);
+ HoodieStorage storage = HoodieStorageUtils.getStorage(storagePath,
HadoopFSUtils.getStorageConf(job));
+ HoodieSchema fileSchema =
HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(storagePath).readSchema(storage,
storagePath);
Review Comment:
Confirmed: the leg reproduced `INT96 is deprecated` on the previous code.
The guard now reads the raw `MessageType` (`ParquetUtils.readMessageType`) and
walks the requested column's group by shape (`typed_value` + `metadata`, any
depth); the `HoodieSchema` walker is deleted. Pinned with an unshredded and a
shredded file both carrying a real INT96 column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
