hudi-agent commented on code in PR #19842:
URL: https://github.com/apache/hudi/pull/19842#discussion_r3951293854
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/HoodieRowDataLanceReader.java:
##########
@@ -204,7 +211,72 @@ public ClosableIterator<RowData>
getRowDataIterator(DataType dataType, HoodieSch
@Override
public HoodieSchema getSchema() {
RowType rowType = HoodieFlinkLanceArrowUtils.toRowType(arrowSchema);
- return HoodieSchemaConverter.convertToSchema(rowType);
+ Map<String, String> metadata = arrowSchema.getCustomMetadata();
+ Set<String> vectorColumnNames = HoodieSchema.parseVectorColumnNames(
+ metadata == null ? null :
metadata.get(HoodieSchema.VECTOR_COLUMNS_METADATA_KEY));
+ if (vectorColumnNames.isEmpty()) {
+ return HoodieSchemaConverter.convertToSchema(rowType);
+ }
+ Map<String, Integer> vectorColumns = new LinkedHashMap<>();
+ vectorColumnNames.forEach(name -> vectorColumns.put(
+ name, vectorSchemaFromField(getTopLevelField(name)).getDimension()));
+ return HoodieSchemaConverter.convertToSchema(rowType, "record",
vectorColumns);
+ }
+
+ private void validateRequestedVectors(HoodieSchema requestedSchema) {
+ for (HoodieSchemaField field :
requestedSchema.getNonNullType().getFields()) {
+ HoodieSchema fieldSchema = field.schema().getNonNullType();
+ if (fieldSchema.getType() != HoodieSchemaType.VECTOR) {
+ continue;
+ }
+ HoodieSchema.Vector expected = (HoodieSchema.Vector) fieldSchema;
+ HoodieSchema.Vector actual =
vectorSchemaFromField(getTopLevelField(field.name()));
+ if (actual.getDimension() != expected.getDimension()
+ || actual.getVectorElementType() != expected.getVectorElementType())
{
+ throw new HoodieValidationException(
+ "Incompatible Lance VECTOR encoding for column '" + field.name()
+ + "': requested " + expected.toTypeDescriptor()
+ + " but file contains " + actual.toTypeDescriptor());
+ }
+ }
+ }
+
+ private Field getTopLevelField(String name) {
+ Field field = arrowSchema.findField(name);
Review Comment:
🤖 Arrow's `Schema.findField` throws `IllegalArgumentException("field %s not
found in %s")` rather than returning null (checked against arrow-vector
18.3.0), so the null branch here is dead and a requested VECTOR column that's
absent from the file now surfaces as a raw IAE instead of the
`HoodieValidationException` the previous stream-based lookup produced. Could
this either catch the IAE or go back to the stream/`findFirst` form?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
