[
https://issues.apache.org/jira/browse/SPARK-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603766#comment-14603766
]
Sudhakar Thota commented on SPARK-4988:
---------------------------------------
Both the spark-sql statements are working.
Here is the listing if some one want to test it. The same is attached in a file
too.
--------
1. Data file :
bash-3.2$ cat t1.txt
1,Barney,10.5
2,Nancy,7.5
3,Tony,4.5
5,Fred,3.5
6,Alok,12.5
7,Jan,23.5
8,Barbara,11.5
9,Mike,6.4
10,Deron,3.7
11,Glenn,9.9
12,Seth,7.8
13,Gerome,4.5
14,Alan,34.5
15,Rohan,33.7
16,clifford,3.5
17,Rosstin,1.5
2. Create table with decimal data type.
CREATE EXTERNAL TABLE user(id INT, name STRING, fico Decimal(4,2)) ROW FORMAT
DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE
LOCATION '/Users/sudhakarthota/tmp';
3. Check if all the data is appearing for select statement.
park-sql> select * from user;
1 Barney 10.5
2 Nancy 7.5
3 Tony 4.5
5 Fred 3.5
6 Alok 12.5
7 Jan 23.5
8 Barbara 11.5
9 Mike 6.4
10 Deron 3.7
11 Glenn 9.9
12 Seth 7.8
13 Gerome 4.5
14 Alan 34.5
15 Rohan 33.7
16 clifford 3.5
17 Rosstin 1.5
Time taken: 0.062 seconds, Fetched 16 row(s)
4. Create table test1 that was failing:
create table test1 as select * from user order by fico limit 10;
spark-sql> create table test1 as select * from user order by fico limit 10;
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted file:///user/hive/warehouse/test1
Time taken: 0.223 seconds
spark-sql>
5. select records.
spark-sql> select * from test1;
17 Rosstin 1.5
16 clifford 3.5
5 Fred 3.5
10 Deron 3.7
3 Tony 4.5
13 Gerome 4.5
9 Mike 6.4
2 Nancy 7.5
12 Seth 7.8
11 Glenn 9.9
Time taken: 0.055 seconds, Fetched 10 row(s)
spark-sql>
6. Do the second test that was failing
spark-sql> select * from user where fico >2;
1 Barney 10.5
2 Nancy 7.5
3 Tony 4.5
5 Fred 3.5
6 Alok 12.5
7 Jan 23.5
8 Barbara 11.5
9 Mike 6.4
10 Deron 3.7
11 Glenn 9.9
12 Seth 7.8
13 Gerome 4.5
14 Alan 34.5
15 Rohan 33.7
16 clifford 3.5
Time taken: 0.061 seconds, Fetched 15 row(s)
spark-sql>
------
Thanks
Sudhakar Thota
> "Create table ..as select ..from..order by .. limit 10" report error when one
> col is a Decimal
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-4988
> URL: https://issues.apache.org/jira/browse/SPARK-4988
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: guowei
> Attachments: spark-4988-1.txt
>
>
> A table 'test' with a decimal type col.
> create table test1 as select * from test order by a limit 10;
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0
> (TID 2, localhost): java.lang.ClassCastException: scala.math.BigDecimal
> cannot be cast to org.apache.spark.sql.catalyst.types.decimal.Decimal
> at
> org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrapperFor$2.apply(HiveInspectors.scala:339)
> at
> org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrapperFor$2.apply(HiveInspectors.scala:339)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1$1.apply(InsertIntoHiveTable.scala:111)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1$1.apply(InsertIntoHiveTable.scala:108)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at
> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:108)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:87)
> at
> org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:87)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:195)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
