[
https://issues.apache.org/jira/browse/SPARK-40351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630001#comment-17630001
]
Dustin Smith edited comment on SPARK-40351 at 11/7/22 8:14 PM:
---------------------------------------------------------------
[~tkhomichuk] For point 3, I think it is related to logical optimizations for
Spark SQL for sum and aggregation according to their [Databricks] paper on
Spark SQL (1995) see section 4.3.2 in the link.
Since it is for optimization decimal operations, I dont think it would be
allowed to overwrite it. This is just based on my thought about their paper and
may be incorrect (take with a grain of salt).
[https://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf]
was (Author: dwsmith1983):
[~tkhomichuk] For point 3, I think it is related to logical optimizations for
Spark SQL for sum and aggregation according to their [Databricks] paper on
Spark SQL see section 4.3.2 in the link.
Since it is for optimization decimal operations, I dont think it would be
allowed to overwrite it. This is just based on my thought about their paper and
may be incorrect (take with a grain of salt).
[https://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf]
> Spark Sum increases the precision of DecimalType arguments by 10
> ----------------------------------------------------------------
>
> Key: SPARK-40351
> URL: https://issues.apache.org/jira/browse/SPARK-40351
> Project: Spark
> Issue Type: Question
> Components: Optimizer
> Affects Versions: 3.2.0
> Reporter: Tymofii
> Priority: Minor
>
> Currently in Spark automatically increases Decimal field by 10 (hard coded
> value) after SUM aggregate operation -
> [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.]
> There are a couple of questions:
> # Why was 10 chosen as default one?
> # Does it make sense to allow the user to override this value via
> configuration?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
