[
https://issues.apache.org/jira/browse/PARQUET-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701541#comment-17701541
]
Alexey Diomin commented on PARQUET-1690:
----------------------------------------
[~gszadovszky] could you review the latest version of pr
[https://github.com/apache/parquet-mr/pull/722]
or do we need to resubmit pr (we can add unit tests, but it will require at
least 1G+ on disk for ci, don't know what the limits for your ci workers)?
_> Removing 1.11.0 target because this is an improvement only and not a
regression since 1.10.0._
it's incorrect, the ticket was created with the incorrect *Type*
what we see in real life: # try to write large blobs with byte arrays
# have overflowed in value in check do we need or not to write min/max
statistics
# writing it in final file + have some incorrect offsets
# have corrupted files and exceptions on read _java.lang.RuntimeException:
corrupted file: the footer index is not within the file: 10687086773_
instead, file with 1 recond in 1GB we have at least = 1GB for data + metadata
(1GB for min + 1GB for max)
file with 1gb blob, in reality, consume 5GB+ on disk and can't be read
reproducible example for spark
https://gist.github.com/REASY/98d80827d53e290d53a93f2bc3b76ab0
> Integer Overflow of BinaryStatistics#isSmallerThan()
> ----------------------------------------------------
>
> Key: PARQUET-1690
> URL: https://issues.apache.org/jira/browse/PARQUET-1690
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.11.0
> Reporter: Xinli Shang
> Assignee: Xinli Shang
> Priority: Major
> Labels: pull-request-available
>
> "(min.length() + max.length()) < size" didn't handle integer overflow
> [https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/statistics/BinaryStatistics.java#L103]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
