voonhous commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3877861642
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveHoodieReaderContext.java:
##########
@@ -148,6 +154,62 @@ private ClosableIterator<ArrayWritable>
getFileRecordIterator(StoragePath filePa
fileSchema = dataSchema;
}
+ // Fail fast on shredded variant columns: this reader hands the file to a
plain
+ // parquet-avro read at the requested {metadata, value} projection, so a
file whose variant
+ // group carries typed_value would come back with silent nulls (the typed
rows keep their
+ // payload in typed_value, which the projection drops). Detection is
shape-based on the
+ // footer schema and anchored on the requested column being a variant, so
plain user structs
+ // of the same shape are left alone. toShreddedReadSchema recurses through
structs, array
+ // elements and map values, matching the row writer, which shreds nested
variants too.
+ // The flagged columns are split by Hive's read column names on the outer
conf, since the
+ // per-file copy below gets requiredSchema's names from setSchemas and
cannot tell the two
+ // apart: a column Hive selected fails as Hive-visible nulls; a column
only requiredSchema
+ // names is there for merging (a CUSTOM merge whose merger is not
projection compatible reads
+ // the whole table schema; a merger can also list it as mandatory) and
fails too, because the
+ // reader materializes it at {metadata, value} and the merger would
consume the nulls. Hive
+ // writes the full name list for `select *` and none for count(*), whose
requested schema is
+ // then empty
(HoodieFileGroupReaderBasedRecordReader.createRequestedSchema), so nothing is
+ // flagged unless merging widens it. A read whose nested column paths
+ // (hive.io.file.readNestedColumn.paths) all miss the shredded group is
not flagged either:
+ // Hive's parquet reader materializes only the paths it is given, and the
mask rewrite below
+ // already handles the compacted projection such a read comes back in.
+ if (isParquetOrOrc && requiredSchema.getType() == HoodieSchemaType.RECORD)
{
+ HoodieSchema shreddedReadSchema =
VariantSchemaUtils.toShreddedReadSchema(requiredSchema, fileSchema);
+ if (shreddedReadSchema != requiredSchema) {
+ List<String> shreddedPaths = new ArrayList<>();
+ collectShreddedVariantPaths(requiredSchema, shreddedReadSchema, "",
shreddedPaths);
+ Configuration conf = storage.getConf().unwrapAs(Configuration.class);
+ Set<String> requestedColumns =
Arrays.stream(HoodieColumnProjectionUtils.getReadColumnNames(conf))
+ .map(name -> name.trim().toLowerCase(Locale.ROOT))
+ .collect(Collectors.toSet());
+ List<String> shreddedColumns =
HoodieColumnProjectionUtils.columnsReadingShreddedPaths(conf, shreddedPaths);
+ Map<Boolean, List<String>> byHiveRequest = shreddedColumns.stream()
+ .collect(Collectors.partitioningBy(requestedColumns::contains));
+ List<String> hiveReads = byHiveRequest.get(true);
+ List<String> mergeOnly = byHiveRequest.get(false);
+ if (!hiveReads.isEmpty()) {
+ throw new HoodieException(String.format(
+ "Column(s) '%s' of %s hold a shredded variant (typed_value
present); the Hive reader "
+ + "cannot reconstruct shredded variants. Read the table with
Spark 4.1+, or "
+ + "rewrite it unshredded (e.g. cluster with "
+ + "hoodie.parquet.variant.write.shredding.enabled=false).",
+ String.join(", ", hiveReads), filePath));
+ }
+ if (!mergeOnly.isEmpty()) {
Review Comment:
Gated. The merge-only throw now checks `isSkipMerge(conf)`, derived as
`HoodieFileGroupReaderBasedRecordReader` derives it:
`hoodie.datasource.merge.type` wins when set, else
`hoodie.realtime.merge.skip`. Under skip-merge the widened columns are still
materialized as nulls, but nothing merges on them and the output converter
projects them away before Hive sees the record, so `select id` keeps working.
The Hive-visible bucket is unchanged. Pinned in `TestHiveHoodieReaderContext`:
both spellings pass, `merge.type=payload_combine` + `merge.skip=true` still
throws (merge.type wins), and `select id, v` under skip-merge still throws.
Fixed in 725f0269ac0d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
