[
https://issues.apache.org/jira/browse/IMPALA-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386961#comment-17386961
]
ASF subversion and git services commented on IMPALA-7087:
---------------------------------------------------------
Commit b3c4ac9054aabcfbf04304b315d1b3103c952228 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b3c4ac9 ]
IMPALA-10814: Fix crash on illegal Parquet file
In IMPALA-7087/IMPALA-8131 we allowed reading decimals with
different precision/scale than table metadata. To allow this
we relaxed some checks against the Parquet file schema.
However we should still add some sanity checks, e.g. ignoring
negative values, require that scale is not greater than precision.
This patch adds these sanity checks. Without these checks we might
hit DCHECKS in Parquet fuzz testing.
Change-Id: Ic6b06ac00e2af8f405f7b2b3c2eb952683821431
Reviewed-on: http://gerrit.cloudera.org:8080/17714
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Impala is unable to read Parquet decimal columns with lower precision/scale
> than table metadata
> -----------------------------------------------------------------------------------------------
>
> Key: IMPALA-7087
> URL: https://issues.apache.org/jira/browse/IMPALA-7087
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: decimal, parquet, ramp-up
> Fix For: Impala 4.1
>
> Attachments: binary_decimal_precision_and_scale_widening.parquet
>
>
> This is similar to IMPALA-2515, except relates to a different precision/scale
> in the file metadata rather than just a mismatch in the bytes used to store
> the data. In a lot of cases we should be able to convert the decimal type on
> the fly to the higher-precision type.
> {noformat}
> ERROR: File '/hdfs/path/000000_0_x_2' column 'alterd_decimal' has an invalid
> type length. Expecting: 11 len in file: 8
> {noformat}
> It would be convenient to allow reading parquet files where the
> precision/scale in the file can be converted to the precision/scale in the
> table metadata without loss of precision.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
