danny0405 commented on code in PR #19831:
URL: https://github.com/apache/hudi/pull/19831#discussion_r3930357642
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/lance/HoodieFlinkLanceArrowUtils.java:
##########
@@ -180,6 +197,36 @@ private static Field toArrowField(String name, LogicalType
type) {
return new Field(name, new FieldType(type.isNullable(), toArrowType(type),
null), children);
}
+ private static Field toArrowField(String name, LogicalType type,
HoodieSchema hoodieSchema) {
+ if (hoodieSchema.getType() != HoodieSchemaType.VECTOR) {
+ return toArrowField(name, type);
+ }
+
+ HoodieSchema.Vector vectorSchema = (HoodieSchema.Vector) hoodieSchema;
+ validateLanceVector(name, vectorSchema);
+ ValidationUtils.checkArgument(type instanceof ArrayType,
+ "VECTOR column '" + name + "' must map to a Flink ARRAY type");
+ ArrayType arrayType = (ArrayType) type;
+ Field elementField = toArrowField("element", arrayType.getElementType());
+ return new Field(
+ name,
+ new FieldType(
+ type.isNullable(),
+ new ArrowType.FixedSizeList(vectorSchema.getDimension()),
Review Comment:
[P1] Add FixedSizeList support to the Flink Lance reader
This changes configured VECTOR columns to FixedSizeListVector, but
HoodieFlinkLanceArrowUtils.readValue still handles every ARRAY by casting the
vector to ListVector, and toLogicalType has no ArrowType.FixedSizeList branch.
A Flink read or update of a Lance file produced here will therefore fail
(ClassCastException on data reads, or HoodieNotSupportedException from
HoodieRowDataLanceReader.getSchema()). Please add FixedSizeList schema/value
conversion, restore VECTOR identity from hoodie.vector.columns, and cover a
writer-to-HoodieRowDataLanceReader round trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
