hudi-agent commented on code in PR #19842:
URL: https://github.com/apache/hudi/pull/19842#discussion_r3933389743
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/HoodieRowDataLanceReader.java:
##########
@@ -204,7 +211,53 @@ public ClosableIterator<RowData>
getRowDataIterator(DataType dataType, HoodieSch
@Override
public HoodieSchema getSchema() {
RowType rowType = HoodieFlinkLanceArrowUtils.toRowType(arrowSchema);
- return HoodieSchemaConverter.convertToSchema(rowType);
+ Map<String, String> metadata = arrowSchema.getCustomMetadata();
+ Set<String> vectorColumnNames = HoodieSchema.parseVectorColumnNames(
+ metadata == null ? null :
metadata.get(HoodieSchema.VECTOR_COLUMNS_METADATA_KEY));
+ if (vectorColumnNames.isEmpty()) {
+ return HoodieSchemaConverter.convertToSchema(rowType);
+ }
+ String vectorColumns = vectorColumnNames.stream()
+ .map(name -> name + ":" +
vectorSchemaFromArrow(getTopLevelField(name)).getDimension())
+ .collect(Collectors.joining(","));
+ return HoodieSchemaConverter.convertToSchema(rowType, "record",
vectorColumns);
+ }
+
+ private void validateRequestedVectors(HoodieSchema requestedSchema) {
+ for (HoodieSchemaField field :
requestedSchema.getNonNullType().getFields()) {
+ HoodieSchema fieldSchema = field.schema().getNonNullType();
+ if (fieldSchema.getType() != HoodieSchemaType.VECTOR) {
+ continue;
+ }
+ HoodieSchema.Vector expected = (HoodieSchema.Vector) fieldSchema;
+ HoodieSchema.Vector actual =
vectorSchemaFromArrow(getTopLevelField(field.name()));
+ if (actual.getDimension() != expected.getDimension()
+ || actual.getVectorElementType() != expected.getVectorElementType())
{
+ throw new HoodieValidationException(
+ "Incompatible Lance VECTOR encoding for column '" + field.name()
+ + "': requested " + expected.toTypeDescriptor()
+ + " but file contains " + actual.toTypeDescriptor());
+ }
+ }
+ }
+
+ private Field getTopLevelField(String name) {
+ return arrowSchema.getFields().stream()
+ .filter(field -> field.getName().equals(name))
+ .findFirst()
+ .orElseThrow(() -> new HoodieValidationException(
+ "Missing Lance column in file schema: " + name));
+ }
+
+ private static HoodieSchema.Vector vectorSchemaFromArrow(Field field) {
Review Comment:
🤖 nit: `vectorSchemaFromArrow` sounds like it returns a schema, but it
actually returns a `HoodieSchema.Vector` type descriptor — could you rename it
to something like `toVectorType` or `vectorTypeFromField` to better reflect
what it produces?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
