[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6814 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-114374392 LGTM, merging this into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113789820 [Test build #35362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35362/console) for PR 6814 at commit [`071a757`](https://github.com/apache/spark/commit/071a7572475ea41423eb5fade27abe06df01508c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113789829 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113777922 [Test build #35362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35362/consoleFull) for PR 6814 at commit [`071a757`](https://github.com/apache/spark/commit/071a7572475ea41423eb5fade27abe06df01508c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113777818 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113777827 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32885075
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -286,6 +288,9 @@ object Decimal {
/** Maximum number of decimal digits a Long can represent */
val MAX_LONG_DIGITS = 18
+ /** Maximum precision a Decimal can support */
+ val MAX_PRECISION = 38
--- End diff --
ok. we can use `MathContext.UNLIMITED` for unlimited precision on scala
decimal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32881564
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -286,6 +288,9 @@ object Decimal {
/** Maximum number of decimal digits a Long can represent */
val MAX_LONG_DIGITS = 18
+ /** Maximum precision a Decimal can support */
+ val MAX_PRECISION = 38
--- End diff --
ok. however, with the customized `MathContext` approach, we can not
determine the needed precision needed for later decimal operations like
multiplication without losing precision reported in the JIRA. Directly using
java decimal to do these operations can avoid this problem as showed in first
commit. Do we want to use it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32881524
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -137,9 +139,9 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def toBigDecimal: BigDecimal = {
if (decimalVal.ne(null)) {
- decimalVal
+ decimalVal(new MathContext(Decimal.MAX_PRECISION,
RoundingMode.HALF_EVEN))
} else {
- BigDecimal(longVal, _scale)
+ BigDecimal(longVal, _scale)(new MathContext(Decimal.MAX_PRECISION,
RoundingMode.HALF_EVEN))
--- End diff --
`_precesion` is the current precision of this decimal. But here we want to
assign a maximum precision that affects in later decimal operation like
multiplication.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32869361
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/types/decimal/DecimalSuite.scala
---
@@ -162,4 +162,9 @@ class DecimalSuite extends SparkFunSuite with
PrivateMethodTester {
assert(new Decimal().set(100L, 10, 0).toUnscaledLong === 100L)
assert(Decimal(Long.MaxValue, 100, 0).toUnscaledLong === Long.MaxValue)
}
+
+ test("accurate precision after multiplication") {
+val decimal = (Decimal(Long.MaxValue, 100, 0) * Decimal(Long.MaxValue,
100, 0)).toJavaBigDecimal
--- End diff --
We can use 38 in this test case
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32869329
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -286,6 +288,9 @@ object Decimal {
/** Maximum number of decimal digits a Long can represent */
val MAX_LONG_DIGITS = 18
+ /** Maximum precision a Decimal can support */
+ val MAX_PRECISION = 38
--- End diff --
Having a short discussion with @marmbrus , we won't going to have fixed
maximum precision in short term, will still support higher even unlimited
precision.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32869203
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -137,9 +139,9 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def toBigDecimal: BigDecimal = {
if (decimalVal.ne(null)) {
- decimalVal
+ decimalVal(new MathContext(Decimal.MAX_PRECISION,
RoundingMode.HALF_EVEN))
} else {
- BigDecimal(longVal, _scale)
+ BigDecimal(longVal, _scale)(new MathContext(Decimal.MAX_PRECISION,
RoundingMode.HALF_EVEN))
--- End diff --
I think we should use the `_precesion` in Decimal object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113641763 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113641669 [Test build #35308 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35308/console) for PR 6814 at commit [`a43bfc3`](https://github.com/apache/spark/commit/a43bfc3d73e20739110469097e13b1d943b99415). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113606420 [Test build #35308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35308/consoleFull) for PR 6814 at commit [`a43bfc3`](https://github.com/apache/spark/commit/a43bfc3d73e20739110469097e13b1d943b99415). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113606195 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113606161 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113605727 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113597204 The style error is from #3347. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113594269 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113594263 [Test build #35300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35300/console) for PR 6814 at commit [`a43bfc3`](https://github.com/apache/spark/commit/a43bfc3d73e20739110469097e13b1d943b99415). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113593225 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113593540 [Test build #35300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35300/consoleFull) for PR 6814 at commit [`a43bfc3`](https://github.com/apache/spark/commit/a43bfc3d73e20739110469097e13b1d943b99415). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113593267 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32850052
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
I think this can be fixed by in `toBigDecimal`:
```
BigDecimal(longVal, _scale, new MathContext(precision,
RoundingMode.HALF_EVEN))
```
Also, in order to have the same behavior as other datebase or Hive, we
should throw an exception if precision is higher than 38.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32848497
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
It can't. But `85070591730234615847396907784232501249` can be handled by
precision 38. As the `MathContext` has only precision 34, we will have scale -4
and get different result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32847892
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
Can Hive support higher precision than 38? I think should match the
behavior in Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32847142
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
However, as decimal type in Hive is based on Java's BigDecimal. As I
tested, Hive can output the accurate result of `select CAST(9223372036854775807
as DECIMAL(38,0)) * CAST(9223372036854775807 as DECIMAL(38,0));`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32844941
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
OK. I just found their results are different due to the `MathContext`. As
@davies said, its precision is 34. Since we ask to print the `unscaledValue`,
so the scale -4 is not applied on the value. So this should not be a bug. Maybe
we don't need to do this modification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32810151
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
`toBigDecimal` just creates scala `BigDecimal` with its information. I
think it is correct. The problem looks like the scala `BigDecimal` produces
wrong result, compared with java `BigDecimal` when doing this multiplication.
To show that, we can create a scala `BigDecimal`. We find that it has
correct precision as same as its underlying java `BigDecimal`:
scala> val d = BigDecimal(Long.MaxValue, 0)
d: scala.math.BigDecimal = 9223372036854775807
scala> d.precision
res16: Int = 19
scala> d.underlying.precision
res17: Int = 19
scala> d.scale
res18: Int = 0
scala> d.underlying.scale
res19: Int = 0
When we multiply two scala `BigDecimal` carrying `Long.MaxValue`, we get
wrong result:
scala> val t = BigDecimal(Long.MaxValue, 0) *
BigDecimal(Long.MaxValue, 0)
t: scala.math.BigDecimal = 8.507059173023461584739690778423250E+37
scala> t.precision
res20: Int = 34
scala> t.scale
res21: Int = -4
scala> t.underlying.unscaledValue.toString
res22: String = 8507059173023461584739690778423250
When we multiply two java `BigDecimal` carrying `Long.MaxValue`, the result
is correct:
scala> val j = d.underlying.multiply(d.underlying)
j: java.math.BigDecimal = 85070591730234615847396907784232501249
scala> j.precision
res23: Int = 38
scala> j.scale
res24: Int = 0
scala> j.toString
res25: String = 85070591730234615847396907784232501249
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113267633 Ah, okay. We should make sure we do exactly the same thing as Hive -- it's possible that Hive also uses this context internally. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113265946 @mateiz We use DECIMAL128 as the MathContext to create BigDecimal, which has precision as 34, it's lower than 38 in Hive. ``` scala> val d = Decimal(2L<<60, 38, 0) d: org.apache.spark.sql.types.Decimal = 2305843009213693952 scala> d * d res0: org.apache.spark.sql.types.Decimal = 5.316911983139663491615228241121378E+36 scala> (d * d).toJavaBigDecimal.unscaledValue res5: java.math.BigInteger = 5316911983139663491615228241121378 // this is wrong ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-113260121 Hive doesn't actually support BigDecimals with precision above 38. Why did you want to add these? It may be okay to add them, but I think the current code works fine for decimals with lower precision. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6814#discussion_r32764893
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -261,7 +261,7 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def - (that: Decimal): Decimal = Decimal(toBigDecimal -
that.toBigDecimal)
- def * (that: Decimal): Decimal = Decimal(toBigDecimal *
that.toBigDecimal)
+ def * (that: Decimal): Decimal =
Decimal(toJavaBigDecimal.multiply(that.toJavaBigDecimal))
--- End diff --
This is kind of workaround, didn't fix the root cause.
The root cause should be in `toBigDecimal`, which doesn't carry on the
information about precision to BigDecimal. Could you fix that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-111815768 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-111808013 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-111808570 [Test build #34882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34882/consoleFull) for PR 6814 at commit [`44c9348`](https://github.com/apache/spark/commit/44c9348dbc6a51d0ac8b99d0280af9849881e67a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-111807945 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6814#issuecomment-111815761 [Test build #34882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34882/console) for PR 6814 at commit [`44c9348`](https://github.com/apache/spark/commit/44c9348dbc6a51d0ac8b99d0280af9849881e67a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class JoinedRow extends InternalRow ` * `class JoinedRow2 extends InternalRow ` * `class JoinedRow3 extends InternalRow ` * `class JoinedRow4 extends InternalRow ` * `class JoinedRow5 extends InternalRow ` * `class JoinedRow6 extends InternalRow ` * `class BaseOrdering extends Ordering[InternalRow] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8359][SQL] Fix incorrect decimal precis...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/6814 [SPARK-8359][SQL] Fix incorrect decimal precision after multiplication JIRA: https://issues.apache.org/jira/browse/SPARK-8359 You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 fix_decimal2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6814.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6814 commit 44c9348dbc6a51d0ac8b99d0280af9849881e67a Author: Liang-Chi Hsieh Date: 2015-06-14T09:58:12Z Fix incorrect decimal precision after multiplication. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
