danny0405 commented on code in PR #19842:
URL: https://github.com/apache/hudi/pull/19842#discussion_r3946338983
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/HoodieRowDataLanceReader.java:
##########
@@ -168,6 +174,7 @@ public ClosableIterator<RowData> getRowDataIterator(
}
public ClosableIterator<RowData> getRowDataIterator(DataType dataType,
HoodieSchema requestedSchema) {
+ validateRequestedVectors(requestedSchema);
Review Comment:
[P2] Preserve the requested VECTOR schema through the COW scan path
This validation is bypassed by `CopyOnWriteInputFormat`: its Lance branch
calls `FormatUtils.getLanceRecordIterator`, which reconstructs
`requestedSchema` with `convertToSchema(selectedDataType.getLogicalType())` and
therefore turns every VECTOR into an ordinary ARRAY. I reproduced this with a
real Lance file containing `embedding:VECTOR(2)` and a `CopyOnWriteInputFormat`
constructed with a table schema declaring `embedding:VECTOR(3)`: the scan
succeeds and returns a two-element array, whereas calling this reader directly
with the same requested VECTOR schema correctly throws
`HoodieValidationException`. Please pass the projected Hudi schema, including
vector dimensions and element types, through that caller (and the other
`FormatUtils` callers), and add an input-format-level mismatch test so normal
scans enforce this check too.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/HoodieRowDataLanceReader.java:
##########
@@ -204,7 +211,53 @@ public ClosableIterator<RowData>
getRowDataIterator(DataType dataType, HoodieSch
@Override
public HoodieSchema getSchema() {
RowType rowType = HoodieFlinkLanceArrowUtils.toRowType(arrowSchema);
- return HoodieSchemaConverter.convertToSchema(rowType);
+ Map<String, String> metadata = arrowSchema.getCustomMetadata();
+ Set<String> vectorColumnNames = HoodieSchema.parseVectorColumnNames(
+ metadata == null ? null :
metadata.get(HoodieSchema.VECTOR_COLUMNS_METADATA_KEY));
+ if (vectorColumnNames.isEmpty()) {
+ return HoodieSchemaConverter.convertToSchema(rowType);
+ }
+ String vectorColumns = vectorColumnNames.stream()
+ .map(name -> name + ":" +
vectorSchemaFromArrow(getTopLevelField(name)).getDimension())
+ .collect(Collectors.joining(","));
+ return HoodieSchemaConverter.convertToSchema(rowType, "record",
vectorColumns);
Review Comment:
[P2] Preserve file column-name casing during VECTOR restoration
The string overload of `HoodieSchemaConverter.convertToSchema` routes these
file names through `VectorColumnParser.parse`, which lowercases them, while
`validateVectorColumns` compares against the original Arrow field names.
Consequently a valid file with a FLOAT vector named `Embedding` and footer
metadata `Embedding:VECTOR(2)` fails `getSchema()` with `VECTOR column
'embedding' does not exist in the table schema.` I reproduced this by writing
the file from an explicitly constructed `HoodieSchema` using
`HoodieRowDataLanceWriter`; the equivalent lowercase field succeeds. This also
affects schema discovery for mixed-case names in Spark-written files. Please
restore the VECTOR fields using their exact file names, without passing them
through the table-option name normalizer, and cover a mixed-case field in the
restoration test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
