voonhous commented on code in PR #19288:
URL: https://github.com/apache/hudi/pull/19288#discussion_r3662819839
##########
hudi-trino/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java:
##########
@@ -192,33 +201,52 @@ public ConnectorPageSource createPageSource(
.withBloomFilter(useParquetBloomFilter(session))
.withVectorizedDecodingEnabled(isParquetVectorizedDecodingEnabled(session))
.build();
- ConnectorPageSource dataPageSource = createPageSource(
- session,
- isBaseFileOnly ? dataColumnHandles :
hudiMetaAndDataColumnHandles,
- hudiSplit,
-
fileSystem.newInputFile(Location.of(hudiBaseFileOpt.get().getPath()),
hudiBaseFileOpt.get().getFileSize()),
- hudiBaseFileOpt.get().getPath(),
- start,
- length,
- OptionalLong.of(hudiBaseFileOpt.get().getFileSize()),
- dataSourceStats,
- sessionOptions,
- timeZone, dynamicFilter, isBaseFileOnly);
-
- SynthesizedColumnHandler synthesizedColumnHandler =
SynthesizedColumnHandler.create(hudiSplit);
+ PrefilledColumnValues prefilledColumnValues =
PrefilledColumnValues.create(hudiSplit);
// Avoid avro serialization if split/filegroup only contains base files
if (isBaseFileOnly) {
return new HudiBaseFileOnlyPageSource(
- dataPageSource,
+ createBaseFilePageSource(session, dataColumnHandles,
hudiSplit, fileSystem, sessionOptions, start, length, dynamicFilter, true),
hiveColumnHandles,
dataColumnHandles,
- synthesizedColumnHandler);
+ prefilledColumnValues);
+ }
+
+ // The merge path below is built around a base-file page source; fail
log-only file slices with a
+ // clear error instead of an opaque NoSuchElementException from
getBaseFile().orElseThrow().
+ // TODO: support log-only file slices by feeding the file-group reader
an empty base page source.
+ if (hudiBaseFileOpt.isEmpty()) {
+ throw new TrinoException(NOT_SUPPORTED, "Hudi splits with log
files but no base file are not supported: "
+ + hudiSplit.getLogFiles().getFirst().getPath());
}
// TODO: Move this into HudiTableHandle
HoodieTableMetaClient metaClient = buildTableMetaClient(
fileSystemFactory.create(session),
hudiTableHandle.getSchemaTableName().toString(), hudiTableHandle.getBasePath());
+ HoodieSchema dataSchema =
+ Optional.ofNullable(hudiTableHandle.getTableSchema())
+ .orElseGet(() -> getLatestTableSchema(metaClient,
hudiTableHandle.getTableName()));
+ TypedProperties readerProps = buildReaderProperties(session,
metaClient);
+
+ // A non-projection-compatible CUSTOM merger makes the file-group
reader demand the FULL table
+ // schema as requiredSchema for this split (it has log files), for the
base and log reads alike.
+ // Expand the read projection to the full schema up front so the base
page source carries every
+ // merge column; the log page sources resolve their columns from the
same expanded handles.
+ List<HiveColumnHandle> readColumnHandles;
+ if (requiresFullSchemaRead(metaClient.getTableConfig(), readerProps)) {
+ log.debug("Expanding the read projection of %s to the full table
schema: the resolved record merger is not projection compatible",
+ hudiTableHandle.getSchemaTableName());
+ readColumnHandles = appendMissingSchemaColumns(dataSchema,
hudiMetaAndDataColumnHandles);
+ }
+ else {
+ // The metastore may lack merge-required columns the table schema
carries (e.g. hive sync with
+ // omit_metadata_fields=true drops _hoodie_operation); recover
them from the already-resolved
+ // schema so the base read is not starved of them.
+ readColumnHandles = appendMissingMergeRequiredColumns(dataSchema,
hudiMetaAndDataColumnHandles, metaClient.getTableConfig());
Review Comment:
Added exactly that: `omitted_ordering_field_mor`, an EVENT_TIME_ORDERING MOR
fixture whose metastore column list omits the `ts` its Avro schema carries.
Row-level and narrow-projection `sum(value) = 199` cases in
`TestHudiNonProjectionCompatibleMerger`, and I verified the pin the way you
framed it: with the line reverted to `hudiMetaAndDataColumnHandles`, both fail
with the missing-column guard error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
