[jira] [Assigned] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31867: --- Assignee: Kent Yao > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Blocker > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} > For patterns that support `SignStyle.EXCEEDS_PAD`, e.g. `y..y`(len >=4), when > using the `NumberPrinterParser` to format it > {code:java} > switch (signStyle) { > case EXCEEDS_PAD: > if (minWidth < 19 && value >= EXCEED_POINTS[minWidth]) { > buf.append(decimalStyle.getPositiveSign()); > } > break; > > > {code} > the `minWidth` == `len(y..y)` > the `EXCEED_POINTS` is > {code:java} > /** > * Array of 10 to the power of n. > */ > static final long[] EXCEED_POINTS = new long[] { > 0L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > }; > {code} > So when the `len(y..y)` is greater than 10, ` ArrayIndexOutOfBoundsException` > will be raised. > And at the caller side, for `from_unixtime`, the exception will be > suppressed and silent data change occurs. for `date_format`, the > `ArrayIndexOutOfBoundsException` will continue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31867: Assignee: Apache Spark > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Apache Spark >Priority: Blocker > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31867: Assignee: (was: Apache Spark) > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Blocker > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org