chong created SPARK-38520:
-----------------------------
Summary: Overflow occurs when reading ANSI day time interval from
CSV file
Key: SPARK-38520
URL: https://issues.apache.org/jira/browse/SPARK-38520
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.3.0
Reporter: chong
*Problem:*
Overflow occurs when reading the following positive intervals, the results
become to negative
interval '106751992' day => INTERVAL '-106751990' DAY
INTERVAL +'+2562047789' hour => INTERVAL '-2562047787' HOUR
interval '153722867281' minute => INTERVAL '-153722867280' MINUTE
*Produce:*
```
// days overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
scala> spark.read.csv(path).show(false)
+------------------------+
|_c0 |
+------------------------+
|interval '106751992' day|
+------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+-------------------------+
|c1 |
+-------------------------+
|INTERVAL '-106751990' DAY|
+-------------------------+
// hour overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
scala> spark.read.csv(path).show(false)
+----------------------------+
|_c0 |
+----------------------------+
|INTERVAL +'+2562047789' hour|
+----------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+---------------------------+
|c1 |
+---------------------------+
|INTERVAL '-2562047787' HOUR|
+---------------------------+
// minute overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.MINUTE,
DayTimeIntervalType.MINUTE))))
scala> spark.read.csv(path).show(false)
+------------------------------+
|_c0 |
+------------------------------+
|interval '153722867281' minute|
+------------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+-------------------------------+
|c1 |
+-------------------------------+
|INTERVAL '-153722867280' MINUTE|
+-------------------------------+
```
*others:*
Also check the negative value is read to positive.
others:
should check the negative also,
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
