[
https://issues.apache.org/jira/browse/ARROW-9177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267058#comment-17267058
]
Steve M. Kim commented on ARROW-9177:
-------------------------------------
I wrote a small program that generates a simplified Parquet file to show this
incompatibility.
[https://github.com/chairmank/arrow-9177-example]
An example file can be downloaded directly from
[https://github.com/chairmank/arrow-9177-example/blob/7685f21ea4acad2d8a834e6890ea9293406da3ee/561120a3094ee4513ba619b518c7a6093fe4e38398219ad172fb75373c3360b8.parquet]
This example file has 10000 rows. By bisection search, I found that the
"Corrupt LZ4 compressed data" error occurred when there were 6528 rows, but not
when there were fewer rows. This is a clue that the problem is somehow related
to data size.
> [C++][Parquet] Tracking issue for cross-implementation LZ4 Parquet
> compression compatibility
> --------------------------------------------------------------------------------------------
>
> Key: ARROW-9177
> URL: https://issues.apache.org/jira/browse/ARROW-9177
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Wes McKinney
> Assignee: Antoine Pitrou
> Priority: Critical
> Fix For: 2.0.0
>
>
> per PARQUET-1878, it seems that there are still problems with our use of LZ4
> compression in the Parquet format. While we should fix this (the Parquet
> specification and our implementation of it), we may need to disable use of
> LZ4 compression until the appropriate compatibility testing can bed one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
