[
https://issues.apache.org/jira/browse/SPARK-38520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chong updated SPARK-38520:
--------------------------
Description:
*Problem:*
Overflow occurs when reading the following positive intervals, the results
become to negative
interval '106751992' day => INTERVAL '-106751990' DAY
INTERVAL +'+2562047789' hour => INTERVAL '-2562047787' HOUR
interval '153722867281' minute => INTERVAL '-153722867280' MINUTE
*Reproduce:*
{code:java}
// days overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
scala> spark.read.csv(path).show(false)
+------------------------+
|_c0 |
+------------------------+
|interval '106751992' day|
+------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+-------------------------+
|c1 |
+-------------------------+
|INTERVAL '-106751990' DAY|
+-------------------------+
// hour overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
scala> spark.read.csv(path).show(false)
+----------------------------+
|_c0 |
+----------------------------+
|INTERVAL +'+2562047789' hour|
+----------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+---------------------------+
|c1 |
+---------------------------+
|INTERVAL '-2562047787' HOUR|
+---------------------------+
// minute overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.MINUTE,
DayTimeIntervalType.MINUTE))))
scala> spark.read.csv(path).show(false)
+------------------------------+
|_c0 |
+------------------------------+
|interval '153722867281' minute|
+------------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+-------------------------------+
|c1 |
+-------------------------------+
|INTERVAL '-153722867280' MINUTE|
+-------------------------------+
{code}
*others:*
Also check the negative value is read to positive.
others:
should check the negative also,
was:
*Problem:*
Overflow occurs when reading the following positive intervals, the results
become to negative
interval '106751992' day => INTERVAL '-106751990' DAY
INTERVAL +'+2562047789' hour => INTERVAL '-2562047787' HOUR
interval '153722867281' minute => INTERVAL '-153722867280' MINUTE
*Reroduce:*
{code}
// days overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
scala> spark.read.csv(path).show(false)
+------------------------+
|_c0 |
+------------------------+
|interval '106751992' day|
+------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+-------------------------+
|c1 |
+-------------------------+
|INTERVAL '-106751990' DAY|
+-------------------------+
// hour overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
scala> spark.read.csv(path).show(false)
+----------------------------+
|_c0 |
+----------------------------+
|INTERVAL +'+2562047789' hour|
+----------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+---------------------------+
|c1 |
+---------------------------+
|INTERVAL '-2562047787' HOUR|
+---------------------------+
// minute overflow
scala> val schema = StructType(Seq(StructField("c1",
DayTimeIntervalType(DayTimeIntervalType.MINUTE,
DayTimeIntervalType.MINUTE))))
scala> spark.read.csv(path).show(false)
+------------------------------+
|_c0 |
+------------------------------+
|interval '153722867281' minute|
+------------------------------+
scala> spark.read.schema(schema).csv(path).show(false)
+-------------------------------+
|c1 |
+-------------------------------+
|INTERVAL '-153722867280' MINUTE|
+-------------------------------+
{code}
*others:*
Also check the negative value is read to positive.
others:
should check the negative also,
> Overflow occurs when reading ANSI day time interval from CSV file
> -----------------------------------------------------------------
>
> Key: SPARK-38520
> URL: https://issues.apache.org/jira/browse/SPARK-38520
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.3.0
> Reporter: chong
> Priority: Major
>
> *Problem:*
> Overflow occurs when reading the following positive intervals, the results
> become to negative
> interval '106751992' day => INTERVAL '-106751990' DAY
> INTERVAL +'+2562047789' hour => INTERVAL '-2562047787' HOUR
> interval '153722867281' minute => INTERVAL '-153722867280' MINUTE
>
> *Reproduce:*
> {code:java}
> // days overflow
> scala> val schema = StructType(Seq(StructField("c1",
> DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))))
> scala> spark.read.csv(path).show(false)
> +------------------------+
> |_c0 |
> +------------------------+
> |interval '106751992' day|
> +------------------------+
> scala> spark.read.schema(schema).csv(path).show(false)
> +-------------------------+
> |c1 |
> +-------------------------+
> |INTERVAL '-106751990' DAY|
> +-------------------------+
> // hour overflow
> scala> val schema = StructType(Seq(StructField("c1",
> DayTimeIntervalType(DayTimeIntervalType.HOUR, DayTimeIntervalType.HOUR))))
> scala> spark.read.csv(path).show(false)
> +----------------------------+
> |_c0 |
> +----------------------------+
> |INTERVAL +'+2562047789' hour|
> +----------------------------+
> scala> spark.read.schema(schema).csv(path).show(false)
> +---------------------------+
> |c1 |
> +---------------------------+
> |INTERVAL '-2562047787' HOUR|
> +---------------------------+
> // minute overflow
> scala> val schema = StructType(Seq(StructField("c1",
> DayTimeIntervalType(DayTimeIntervalType.MINUTE,
> DayTimeIntervalType.MINUTE))))
> scala> spark.read.csv(path).show(false)
> +------------------------------+
> |_c0 |
> +------------------------------+
> |interval '153722867281' minute|
> +------------------------------+
> scala> spark.read.schema(schema).csv(path).show(false)
> +-------------------------------+
> |c1 |
> +-------------------------------+
> |INTERVAL '-153722867280' MINUTE|
> +-------------------------------+
> {code}
>
> *others:*
> Also check the negative value is read to positive.
>
> others:
> should check the negative also,
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
