Gert Hulselmans created ARROW-12904:
---------------------------------------
Summary: [Rust] Unable to load Feather v2 files created by pyarrow
and pandas.
Key: ARROW-12904
URL: https://issues.apache.org/jira/browse/ARROW-12904
Project: Apache Arrow
Issue Type: Bug
Components: Rust
Affects Versions: 4.0.1
Environment: Ubuntu 20.04
Reporter: Gert Hulselmans
Assignee: Joris Van den Bossche
arrow-rs seems unable to load Feather v2 files created by pyarrow (and pandas),
while it can read Feather v2 created by itself.
More info at:
[https://github.com/apache/arrow-rs/issues/286]
Any idea what is missing in the Rust implementation (missing part of the spec?)?
{code:java}
More details: in both files, I am getting the following:
Reading Utf8
field_node: FieldNode { length: 7, null_count: 0 }
offset buffer: Buffer { offset: 200, length: 55 }
offsets: [32, 0, 407708164, 545407072, 8388608, 67108864, 134217728, 201326592]
values buffer: Buffer { offset: 256, length: 51 }
offsets[0] != 0 indicates a problem: offsets are expected to start from zero on
any array with offsets.
offsets[i+1] < offsets[i+1] for some i, which indicates a problem: offsets are
expected to be monotonically increasing
I do not have a root cause yet, these are just observations.
{code}
https://github.com/apache/arrow-rs/issues/286#issuecomment-839524898
In the attachment the following files can be found.
{{}}
{code:java}
test_pandas.feather: Original Feather file
test_arrow.feather: loading test_pandas.feather with pyarrow and saving with
pyarrow: df_pa = pa.feather.read_feather('test_pandas.feather')
test_polars.feather: Loading test_pandas.feather with pyarrow and saving with
polars (only this one can be read by arrow-rs)
test_pandas_from_polars.feather: Loading test_polars.feather with polars and
using the to_pandas option.
{code}
[^test_feather_file.zip]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
