yihua opened a new issue, #866:
URL: https://github.com/apache/hudi-rs/issues/866
### Describe the bug
Two shapes of embedding column fail on the write path:
1. A `list<float32>` column appends fine, but the next `upsert` on the table
is rejected with `upsert batch schema does not match the current table schema`.
The table schema is derived from the Avro schema in commit metadata, which
names the list item `element`, while PyArrow (and most Arrow producers) name it
`item`. `align_batch_to_schema` compares `DataType` with `==`, which includes
nested field names, so the batch never matches the table it was just written to.
2. A `FixedSizeList` column is rejected on `append` with `cannot map Arrow
type FixedSizeList(...) to Avro for commit metadata schema`.
### To Reproduce
```python
import numpy as np, pyarrow as pa
from hudi import HudiTable
emb = np.random.default_rng(1).random((3, 4), dtype=np.float32)
tb = pa.table({"id": ["a", "b", "c"], "embedding": pa.array(emb.tolist(),
pa.list_(pa.float32())), "v": [1, 1, 1]})
t = HudiTable.create("/tmp/t", record_key="id", ordering_field="v")
t.append(tb)
t.upsert(tb) # Data type error: upsert batch schema does not match the
current table schema
```
Replace the column with `pa.FixedSizeListArray.from_arrays(..., 4)` and
`append` fails instead.
### Expected behavior
Lists whose only difference from the table schema is a nested field name are
accepted and written in the table's shape; fixed-size lists are written as
lists, which is the only shape Avro and the readers have for them.
### Environment
main at 4dd65422 (after #666), macOS arm64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
