GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/23232
[SPARK-26233][SQL][BACKPORT-2.4] CheckOverflow when encoding a decimal value
When we encode a Decimal from external source we don't check for overflow.
That method is useful not only in order to enforce that we can represent the
correct value in the specified range, but it also changes the underlying data
to the right precision/scale. Since in our code generation we assume that a
decimal has exactly the same precision and scale of its data type, missing to
enforce it can lead to corrupted output/results when there are subsequent
transformations.
added UT
Closes #23210 from mgaido91/SPARK-26233.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark SPARK-26233_2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23232.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23232
commit 821db4854c0e685aac3168da75a1c839681dbfc4
Author: Marco Gaido
Date: 2018-12-04T18:33:27Z
[SPARK-26233][SQL] CheckOverflow when encoding a decimal value
When we encode a Decimal from external source we don't check for overflow.
That method is useful not only in order to enforce that we can represent the
correct value in the specified range, but it also changes the underlying data
to the right precision/scale. Since in our code generation we assume that a
decimal has exactly the same precision and scale of its data type, missing to
enforce it can lead to corrupted output/results when there are subsequent
transformations.
added UT
Closes #23210 from mgaido91/SPARK-26233.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
