cshuo commented on code in PR #19842:
URL: https://github.com/apache/hudi/pull/19842#discussion_r3947539994
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/HoodieRowDataLanceReader.java:
##########
@@ -168,6 +174,7 @@ public ClosableIterator<RowData> getRowDataIterator(
}
public ClosableIterator<RowData> getRowDataIterator(DataType dataType,
HoodieSchema requestedSchema) {
+ validateRequestedVectors(requestedSchema);
Review Comment:
Fixed. The COW path now passes the projected HoodieSchema to the Lance
reader, with a mismatch test added.
I audited the remaining direct FormatUtils.getLanceRecordIterator callers as
well. They belong to CDC base-file reading paths, which require coordinated
VECTOR handling for both Lance and Parquet. I’ve tracked that separately in
#19857 to keep this PR scoped to the regular Lance reader path.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/HoodieRowDataLanceReader.java:
##########
@@ -204,7 +211,53 @@ public ClosableIterator<RowData>
getRowDataIterator(DataType dataType, HoodieSch
@Override
public HoodieSchema getSchema() {
RowType rowType = HoodieFlinkLanceArrowUtils.toRowType(arrowSchema);
- return HoodieSchemaConverter.convertToSchema(rowType);
+ Map<String, String> metadata = arrowSchema.getCustomMetadata();
+ Set<String> vectorColumnNames = HoodieSchema.parseVectorColumnNames(
+ metadata == null ? null :
metadata.get(HoodieSchema.VECTOR_COLUMNS_METADATA_KEY));
+ if (vectorColumnNames.isEmpty()) {
+ return HoodieSchemaConverter.convertToSchema(rowType);
+ }
+ String vectorColumns = vectorColumnNames.stream()
+ .map(name -> name + ":" +
vectorSchemaFromArrow(getTopLevelField(name)).getDimension())
+ .collect(Collectors.joining(","));
+ return HoodieSchemaConverter.convertToSchema(rowType, "record",
vectorColumns);
Review Comment:
Fixed. VECTOR column names now preserve their original casing and are
matched case-sensitively. Mixed-case tests have been added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
