[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426351#comment-15426351
]
Abdeali Kothari commented on SPARK-5456:
For the record, I get this error in 1.4.1 too.
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Assignee: Adrian Wang
>Priority: Blocker
> Fix For: 1.3.2, 1.4.0
>
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383392#comment-14383392
]
Karthik G commented on SPARK-5456:
--
This is a blocker when using Spark with Databases which have Decimal / Big
Decimal columns. Is there a workaround?
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386628#comment-14386628
]
Karthik Gorthi commented on SPARK-5456:
---
One workaround we followed is to convert all Decimal Datatype columns to
integer (truncated).
But, this is not possible when the application needs to connect to third-party
databases were the datatype is obviously not under our control.
So, this is a serious bug, IMHO, which prohibits using Spark in cases, unless
there is a workaround?
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387904#comment-14387904
]
Kuldeep commented on SPARK-5456:
[~karthikg01]
1) Switch to hive context, I am not trying to deride the plain sql context, but
the hive context is just better tested and has a well defined syntax borrowed
from hive.
2) Even in hive context i have faced problems with bigdecimals, so like your
workaround i also convert bigdecimals to a double (not int). And for all
practical purposes it is more than enough. I have not seem many datasources
with those types. rdbms maps `NUMERIC` type to bigdecimal in jdbc but you can
always workaround this by have a simple map transformation before you register
it to sql context.
2 cents.
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711690#comment-14711690
]
Brandon Bradley commented on SPARK-5456:
I'm still experiencing this in 1.4.0 and 1.4.1. I believe the fix for it should
be in 1.4.1.
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Assignee: Adrian Wang
>Priority: Blocker
> Fix For: 1.3.2, 1.4.0
>
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516195#comment-14516195
]
Yi Zhang commented on SPARK-5456:
-
I got same issue. When I access Postgresql based on Spark SQL, and the code is
as below:
dataFrame.registerTempTable("Employees")
val emps = sqlContext.sql("select name, sum(salary) from Employees group by
name")
monitor {
emps.take(10)
.map(row => (row.getString(0), row.getDecimal(1)))
.foreach(println)
}
The type of salary column in data table of Postgresql is numeric(10, 2).
Running it, then it throws the following exception:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to
stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost
task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ClassCastException:
java.math.BigDecimal cannot be cast to org.apache.spark.sql.types.Decimal
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Priority: Blocker
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520917#comment-14520917
]
Adrian Wang commented on SPARK-5456:
Hi, I just tried the code in the description of this issue with latest master,
and got not errors.
Can you guys confirm that?
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Priority: Blocker
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521100#comment-14521100
]
Apache Spark commented on SPARK-5456:
-
User 'adrian-wang' has created a pull request for this issue:
https://github.com/apache/spark/pull/5803
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Priority: Blocker
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521109#comment-14521109
]
Adrian Wang commented on SPARK-5456:
I created a pull request for [~zhangyi]
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Priority: Blocker
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[
https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533039#comment-14533039
]
Yin Huai commented on SPARK-5456:
-
Seems it is not in 1.2 branch. So I am removing the fix version "1.2.3".
> Decimal Type comparison issue
> -
>
> Key: SPARK-5456
> URL: https://issues.apache.org/jira/browse/SPARK-5456
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kuldeep
>Assignee: Adrian Wang
>Priority: Blocker
> Fix For: 1.3.2, 1.4.0
>
>
> Not quite able to figure this out but here is a junit test to reproduce this,
> in JavaAPISuite.java
> {code:title=DecimalBug.java}
> @Test
> public void decimalQueryTest() {
> List decimalTable = new ArrayList();
> decimalTable.add(RowFactory.create(new BigDecimal("1"), new
> BigDecimal("2")));
> decimalTable.add(RowFactory.create(new BigDecimal("3"), new
> BigDecimal("4")));
> JavaRDD rows = sc.parallelize(decimalTable);
> List fields = new ArrayList(7);
> fields.add(DataTypes.createStructField("a",
> DataTypes.createDecimalType(), true));
> fields.add(DataTypes.createStructField("b",
> DataTypes.createDecimalType(), true));
> sqlContext.applySchema(rows.rdd(),
> DataTypes.createStructType(fields)).registerTempTable("foo");
> Assert.assertEquals(sqlContext.sql("select * from foo where a >
> 0").collectAsList(), decimalTable);
> }
> {code}
> Fails with
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.spark.sql.types.Decimal
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
