Davies Liu created SPARK-17230:
----------------------------------
Summary: Writing decimal to csv will result empty string if the
decimal exceeds (20, 18)
Key: SPARK-17230
URL: https://issues.apache.org/jira/browse/SPARK-17230
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.0, 1.6.2
Reporter: Davies Liu
Assignee: Davies Liu
{code}
// file content
spark.read.csv("/mnt/djiang/test-case.csv").show
// read in as string and create temp view
spark.read.csv("/mnt/djiang/test-case.csv").createOrReplaceTempView("test")
// confirm schema
spark.table("test").printSchema
// apply decimal calculation, confirm the result is correct
spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) from
test").show(false)
// run the same query, and write out as csv
spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) from
test").write.csv("/mnt/djiang/test-case-result")
// show the content of the result file, particularly, for number exceeded
decimal(20, 18), the csv is not writing anything or failing silently
spark.read.csv("/mnt/djiang/test-case-result").show
+------+
| _c0|
+------+
| 1|
| 10|
| 100|
| 1000|
| 10000|
|100000|
+------+
root
|-- _c0: string (nullable = true)
+------+-------------------------------------------------------------------------------------------------------------------------------------------------+
|_c0 |(CAST(CAST(CAST(CAST(_c0 AS DECIMAL(20,0)) AS BIGINT) AS DECIMAL(20,0))
AS DECIMAL(38,18)) * CAST(CAST(1.0 AS DECIMAL(38,18)) AS DECIMAL(38,18)))|
+------+-------------------------------------------------------------------------------------------------------------------------------------------------+
|1 |1.000000000000000000 |
|10 |10.000000000000000000 |
|100 |100.000000000000000000 |
|1000 |1000.000000000000000000 |
|10000 |10000.000000000000000000 |
|100000|100000.000000000000000000 |
+------+-------------------------------------------------------------------------------------------------------------------------------------------------+
+------+--------------------+
| _c0| _c1|
+------+--------------------+
| 1|1.000000000000000000|
| 10|10.00000000000000...|
| 100| |
| 1000| |
| 10000| |
|100000| |
+------+--------------------+
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
