[
https://issues.apache.org/jira/browse/SPARK-40351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600572#comment-17600572
]
Tymofii commented on SPARK-40351:
---------------------------------
# Not sure I understood why you showed those lines of code.
# For example, the issue we faced is as follows. Source tables have decimal
fields with the certain precision defined during the table creation. There are
number of queries, which are used to extract and transform the data from those
source tables and load it to the target one, which also has a decimal field
with the same precision as in the sources tables. So the users knows for sure,
that summing values in the source decimal fields may not result in exceeding
the target table field precision. Currently they have to add explicit casting
after SUM function to comply with the target table definition since our ETL
flow would fail. It may be not very convenient if there are multiple queries.
So they could disable automatic increase of the precision in this case for
example.
# Another question - what is the rationale behind the number 10?
> Spark Sum increases the precision of DecimalType arguments by 10
> ----------------------------------------------------------------
>
> Key: SPARK-40351
> URL: https://issues.apache.org/jira/browse/SPARK-40351
> Project: Spark
> Issue Type: Question
> Components: Optimizer
> Affects Versions: 3.2.0
> Reporter: Tymofii
> Priority: Minor
>
> Currently in Spark automatically increases Decimal field by 10 (hard coded
> value) after SUM aggregate operation -
> [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.]
> There are a couple of questions:
> # Why was 10 chosen as default one?
> # Does it make sense to allow the user to override this value via
> configuration?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
