[
https://issues.apache.org/jira/browse/ARROW-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369563#comment-17369563
]
Gert Hulselmans commented on ARROW-12904:
-----------------------------------------
Looks like it was caused by lz4 compression used in Feather file that arrow-rs
does not detect properly.
> [Rust] Unable to load Feather v2 files created by pyarrow and pandas.
> ---------------------------------------------------------------------
>
> Key: ARROW-12904
> URL: https://issues.apache.org/jira/browse/ARROW-12904
> Project: Apache Arrow
> Issue Type: Bug
> Components: Rust
> Affects Versions: 4.0.1
> Environment: Ubuntu 20.04
> Reporter: Gert Hulselmans
> Assignee: Joris Van den Bossche
> Priority: Major
>
> arrow-rs seems unable to load Feather v2 files created by pyarrow (and
> pandas), while it can read Feather v2 created by itself.
> More info at:
> [https://github.com/apache/arrow-rs/issues/286]
>
> Any idea what is missing in the Rust implementation (missing part of the
> spec?)?
>
> {code:java}
> More details: in both files, I am getting the following:
> Reading Utf8
> field_node: FieldNode { length: 7, null_count: 0 }
> offset buffer: Buffer { offset: 200, length: 55 }
> offsets: [32, 0, 407708164, 545407072, 8388608, 67108864, 134217728,
> 201326592]
> values buffer: Buffer { offset: 256, length: 51 }
> offsets[0] != 0 indicates a problem: offsets are expected to start from zero
> on any array with offsets.
> offsets[i+1] < offsets[i+1] for some i, which indicates a problem: offsets
> are expected to be monotonically increasing
> I do not have a root cause yet, these are just observations.
> {code}
> https://github.com/apache/arrow-rs/issues/286#issuecomment-839524898
>
> In the attachment the following files can be found.
> {{}}
> {code:java}
> test_pandas.feather: Original Feather file
> test_arrow.feather: loading test_pandas.feather with pyarrow and saving with
> pyarrow: df_pa = pa.feather.read_feather('test_pandas.feather')
> test_polars.feather: Loading test_pandas.feather with pyarrow and saving
> with polars (only this one can be read by arrow-rs)
> test_pandas_from_polars.feather: Loading test_polars.feather with polars and
> using the to_pandas option.
>
>
> {code}
>
> [^test_feather_file.zip]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
