[jira] [Commented] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126654#comment-17126654 ] Apache Spark commented on SPARK-31867: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28736 > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Blocker > Fix For: 3.0.0 > > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} > For patterns that support `SignStyle.EXCEEDS_PAD`, e.g. `y..y`(len >=4), when > using the `NumberPrinterParser` to format it > {code:java} > switch (signStyle) { > case EXCEEDS_PAD: > if (minWidth < 19 && value >= EXCEED_POINTS[minWidth]) { > buf.append(decimalStyle.getPositiveSign()); > } > break; > > > {code} > the `minWidth` == `len(y..y)` > the `EXCEED_POINTS` is > {code:java} > /** > * Array of 10 to the power of n. > */ > static final long[] EXCEED_POINTS = new long[] { > 0L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > }; > {code} > So when the `len(y..y)` is greater than 10, ` ArrayIndexOutOfBoundsException` > will be raised. > And at the caller side, for `from_unixtime`, the exception will be > suppressed and silent data change occurs. for `date_format`, the > `ArrayIndexOutOfBoundsException` will continue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126652#comment-17126652 ] Apache Spark commented on SPARK-31867: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28736 > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Blocker > Fix For: 3.0.0 > > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} > For patterns that support `SignStyle.EXCEEDS_PAD`, e.g. `y..y`(len >=4), when > using the `NumberPrinterParser` to format it > {code:java} > switch (signStyle) { > case EXCEEDS_PAD: > if (minWidth < 19 && value >= EXCEED_POINTS[minWidth]) { > buf.append(decimalStyle.getPositiveSign()); > } > break; > > > {code} > the `minWidth` == `len(y..y)` > the `EXCEED_POINTS` is > {code:java} > /** > * Array of 10 to the power of n. > */ > static final long[] EXCEED_POINTS = new long[] { > 0L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > }; > {code} > So when the `len(y..y)` is greater than 10, ` ArrayIndexOutOfBoundsException` > will be raised. > And at the caller side, for `from_unixtime`, the exception will be > suppressed and silent data change occurs. for `date_format`, the > `ArrayIndexOutOfBoundsException` will continue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120442#comment-17120442 ] Apache Spark commented on SPARK-31867: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28684 > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Blocker > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} > For patterns that support `SignStyle.EXCEEDS_PAD`, e.g. `y..y`(len >=4), when > using the `NumberPrinterParser` to format it > {code:java} > switch (signStyle) { > case EXCEEDS_PAD: > if (minWidth < 19 && value >= EXCEED_POINTS[minWidth]) { > buf.append(decimalStyle.getPositiveSign()); > } > break; > > > {code} > the `minWidth` == `len(y..y)` > the `EXCEED_POINTS` is > {code:java} > /** > * Array of 10 to the power of n. > */ > static final long[] EXCEED_POINTS = new long[] { > 0L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > }; > {code} > So when the `len(y..y)` is greater than 10, ` ArrayIndexOutOfBoundsException` > will be raised. > And at the caller side, for `from_unixtime`, the exception will be > suppressed and silent data change occurs. for `date_format`, the > `ArrayIndexOutOfBoundsException` will continue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120441#comment-17120441 ] Apache Spark commented on SPARK-31867: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28684 > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Blocker > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} > For patterns that support `SignStyle.EXCEEDS_PAD`, e.g. `y..y`(len >=4), when > using the `NumberPrinterParser` to format it > {code:java} > switch (signStyle) { > case EXCEEDS_PAD: > if (minWidth < 19 && value >= EXCEED_POINTS[minWidth]) { > buf.append(decimalStyle.getPositiveSign()); > } > break; > > > {code} > the `minWidth` == `len(y..y)` > the `EXCEED_POINTS` is > {code:java} > /** > * Array of 10 to the power of n. > */ > static final long[] EXCEED_POINTS = new long[] { > 0L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > 1000L, > 1L, > 10L, > 100L, > }; > {code} > So when the `len(y..y)` is greater than 10, ` ArrayIndexOutOfBoundsException` > will be raised. > And at the caller side, for `from_unixtime`, the exception will be > suppressed and silent data change occurs. for `date_format`, the > `ArrayIndexOutOfBoundsException` will continue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31867) Fix silent data change for datetime formatting
[ https://issues.apache.org/jira/browse/SPARK-31867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119389#comment-17119389 ] Apache Spark commented on SPARK-31867: -- User 'yaooqinn' has created a pull request for this issue: https://github.com/apache/spark/pull/28673 > Fix silent data change for datetime formatting > --- > > Key: SPARK-31867 > URL: https://issues.apache.org/jira/browse/SPARK-31867 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Blocker > > {code:java} > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > NULL > spark-sql> set spark.sql.legacy.timeParserPolicy=legacy; > spark.sql.legacy.timeParserPolicy legacy > spark-sql> select from_unixtime(1, 'yyy-MM-dd'); > 0001970-01-01 > spark-sql> > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org