[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426351#comment-15426351 ] Abdeali Kothari commented on SPARK-5456: For the record, I get this error in 1.4.1 too. > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Assignee: Adrian Wang >Priority: Blocker > Fix For: 1.3.2, 1.4.0 > > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383392#comment-14383392 ] Karthik G commented on SPARK-5456: -- This is a blocker when using Spark with Databases which have Decimal / Big Decimal columns. Is there a workaround? > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386628#comment-14386628 ] Karthik Gorthi commented on SPARK-5456: --- One workaround we followed is to convert all Decimal Datatype columns to integer (truncated). But, this is not possible when the application needs to connect to third-party databases were the datatype is obviously not under our control. So, this is a serious bug, IMHO, which prohibits using Spark in cases, unless there is a workaround? > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387904#comment-14387904 ] Kuldeep commented on SPARK-5456: [~karthikg01] 1) Switch to hive context, I am not trying to deride the plain sql context, but the hive context is just better tested and has a well defined syntax borrowed from hive. 2) Even in hive context i have faced problems with bigdecimals, so like your workaround i also convert bigdecimals to a double (not int). And for all practical purposes it is more than enough. I have not seem many datasources with those types. rdbms maps `NUMERIC` type to bigdecimal in jdbc but you can always workaround this by have a simple map transformation before you register it to sql context. 2 cents. > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711690#comment-14711690 ] Brandon Bradley commented on SPARK-5456: I'm still experiencing this in 1.4.0 and 1.4.1. I believe the fix for it should be in 1.4.1. > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Assignee: Adrian Wang >Priority: Blocker > Fix For: 1.3.2, 1.4.0 > > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516195#comment-14516195 ] Yi Zhang commented on SPARK-5456: - I got same issue. When I access Postgresql based on Spark SQL, and the code is as below: dataFrame.registerTempTable("Employees") val emps = sqlContext.sql("select name, sum(salary) from Employees group by name") monitor { emps.take(10) .map(row => (row.getString(0), row.getDecimal(1))) .foreach(println) } The type of salary column in data table of Postgresql is numeric(10, 2). Running it, then it throws the following exception: Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.spark.sql.types.Decimal > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Priority: Blocker > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520917#comment-14520917 ] Adrian Wang commented on SPARK-5456: Hi, I just tried the code in the description of this issue with latest master, and got not errors. Can you guys confirm that? > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Priority: Blocker > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521100#comment-14521100 ] Apache Spark commented on SPARK-5456: - User 'adrian-wang' has created a pull request for this issue: https://github.com/apache/spark/pull/5803 > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Priority: Blocker > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521109#comment-14521109 ] Adrian Wang commented on SPARK-5456: I created a pull request for [~zhangyi] > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Priority: Blocker > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5456) Decimal Type comparison issue
[ https://issues.apache.org/jira/browse/SPARK-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533039#comment-14533039 ] Yin Huai commented on SPARK-5456: - Seems it is not in 1.2 branch. So I am removing the fix version "1.2.3". > Decimal Type comparison issue > - > > Key: SPARK-5456 > URL: https://issues.apache.org/jira/browse/SPARK-5456 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kuldeep >Assignee: Adrian Wang >Priority: Blocker > Fix For: 1.3.2, 1.4.0 > > > Not quite able to figure this out but here is a junit test to reproduce this, > in JavaAPISuite.java > {code:title=DecimalBug.java} > @Test > public void decimalQueryTest() { > List decimalTable = new ArrayList(); > decimalTable.add(RowFactory.create(new BigDecimal("1"), new > BigDecimal("2"))); > decimalTable.add(RowFactory.create(new BigDecimal("3"), new > BigDecimal("4"))); > JavaRDD rows = sc.parallelize(decimalTable); > List fields = new ArrayList(7); > fields.add(DataTypes.createStructField("a", > DataTypes.createDecimalType(), true)); > fields.add(DataTypes.createStructField("b", > DataTypes.createDecimalType(), true)); > sqlContext.applySchema(rows.rdd(), > DataTypes.createStructType(fields)).registerTempTable("foo"); > Assert.assertEquals(sqlContext.sql("select * from foo where a > > 0").collectAsList(), decimalTable); > } > {code} > Fails with > java.lang.ClassCastException: java.math.BigDecimal cannot be cast to > org.apache.spark.sql.types.Decimal -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org