[jira] [Created] (SPARK-31562) Update ExpressionDescription for substring, current_date, and current_timestamp
Takeshi Yamamuro created SPARK-31562: Summary: Update ExpressionDescription for substring, current_date, and current_timestamp Key: SPARK-31562 URL: https://issues.apache.org/jira/browse/SPARK-31562 Project: Spark Issue Type: Improvement Components: Documentation, SQL Affects Versions: 3.0.0 Reporter: Takeshi Yamamuro -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31562) Update ExpressionDescription for substring, current_date, and current_timestamp
[
https://issues.apache.org/jira/browse/SPARK-31562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro updated SPARK-31562:
-
Description:
This PR intends to add entries for substring, current_date, and
current_timestamp in the SQL built-in function documents. Specifically, the
entries are as follows;
SELECT current_date;
SELECT current_timestamp;
SELECT substring('abcd' FROM 1);
SELECT substring('abcd' FROM 1 FOR 2);
> Update ExpressionDescription for substring, current_date, and
> current_timestamp
> ---
>
> Key: SPARK-31562
> URL: https://issues.apache.org/jira/browse/SPARK-31562
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, SQL
>Affects Versions: 3.0.0
>Reporter: Takeshi Yamamuro
>Priority: Major
>
> This PR intends to add entries for substring, current_date, and
> current_timestamp in the SQL built-in function documents. Specifically, the
> entries are as follows;
> SELECT current_date;
> SELECT current_timestamp;
> SELECT substring('abcd' FROM 1);
> SELECT substring('abcd' FROM 1 FOR 2);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31562) Update ExpressionDescription for substring, current_date, and current_timestamp
[
https://issues.apache.org/jira/browse/SPARK-31562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro updated SPARK-31562:
-
Description:
This jira intends to add entries for substring, current_date, and
current_timestamp in the SQL built-in function documents. Specifically, the
entries are as follows;
SELECT current_date;
SELECT current_timestamp;
SELECT substring('abcd' FROM 1);
SELECT substring('abcd' FROM 1 FOR 2);
was:
This PR intends to add entries for substring, current_date, and
current_timestamp in the SQL built-in function documents. Specifically, the
entries are as follows;
SELECT current_date;
SELECT current_timestamp;
SELECT substring('abcd' FROM 1);
SELECT substring('abcd' FROM 1 FOR 2);
> Update ExpressionDescription for substring, current_date, and
> current_timestamp
> ---
>
> Key: SPARK-31562
> URL: https://issues.apache.org/jira/browse/SPARK-31562
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, SQL
>Affects Versions: 3.0.0
>Reporter: Takeshi Yamamuro
>Priority: Major
>
> This jira intends to add entries for substring, current_date, and
> current_timestamp in the SQL built-in function documents. Specifically, the
> entries are as follows;
> SELECT current_date;
> SELECT current_timestamp;
> SELECT substring('abcd' FROM 1);
> SELECT substring('abcd' FROM 1 FOR 2);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31562) Update ExpressionDescription for substring, current_date, and current_timestamp
[
https://issues.apache.org/jira/browse/SPARK-31562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092128#comment-17092128
]
Takeshi Yamamuro commented on SPARK-31562:
--
I'm working on https://github.com/apache/spark/pull/28342
> Update ExpressionDescription for substring, current_date, and
> current_timestamp
> ---
>
> Key: SPARK-31562
> URL: https://issues.apache.org/jira/browse/SPARK-31562
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, SQL
>Affects Versions: 3.0.0
>Reporter: Takeshi Yamamuro
>Priority: Major
>
> This jira intends to add entries for substring, current_date, and
> current_timestamp in the SQL built-in function documents. Specifically, the
> entries are as follows;
> SELECT current_date;
> SELECT current_timestamp;
> SELECT substring('abcd' FROM 1);
> SELECT substring('abcd' FROM 1 FOR 2);
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31563) Failure of InSet.sql for UTF8String collection
Maxim Gekk created SPARK-31563:
--
Summary: Failure of InSet.sql for UTF8String collection
Key: SPARK-31563
URL: https://issues.apache.org/jira/browse/SPARK-31563
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.5, 3.0.0, 3.1.0
Reporter: Maxim Gekk
The InSet expression works on collections of internal Catalyst's types. We can
see this in the optimization when In is replaced by InSet, and In's collection
is evaluated to internal Catalyst's values:
[https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L254]
{code:scala}
if (newList.length > SQLConf.get.optimizerInSetConversionThreshold) {
val hSet = newList.map(e => e.eval(EmptyRow))
InSet(v, HashSet() ++ hSet)
}
{code}
The code existed before the optimization
https://github.com/apache/spark/pull/25754 that made another wrong assumption
about collection types.
If InSet accepts only internal Catalyst's types, the following code shouldn't
fail:
{code:scala}
InSet(Literal("a"), Set("a", "b").map(UTF8String.fromString)).sql
{code}
but it fails with the exception:
{code}
Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a
java.lang.RuntimeException: Unsupported literal type class
org.apache.spark.unsafe.types.UTF8String a
at
org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:88)
at
org.apache.spark.sql.catalyst.expressions.InSet.$anonfun$sql$2(predicates.scala:522)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31563) Failure of InSet.sql for UTF8String collection
[
https://issues.apache.org/jira/browse/SPARK-31563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092168#comment-17092168
]
Maxim Gekk commented on SPARK-31563:
I am working on the issue
> Failure of InSet.sql for UTF8String collection
> --
>
> Key: SPARK-31563
> URL: https://issues.apache.org/jira/browse/SPARK-31563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.5, 3.0.0, 3.1.0
>Reporter: Maxim Gekk
>Priority: Major
>
> The InSet expression works on collections of internal Catalyst's types. We
> can see this in the optimization when In is replaced by InSet, and In's
> collection is evaluated to internal Catalyst's values:
> [https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L254]
> {code:scala}
> if (newList.length > SQLConf.get.optimizerInSetConversionThreshold) {
> val hSet = newList.map(e => e.eval(EmptyRow))
> InSet(v, HashSet() ++ hSet)
> }
> {code}
> The code existed before the optimization
> https://github.com/apache/spark/pull/25754 that made another wrong assumption
> about collection types.
> If InSet accepts only internal Catalyst's types, the following code shouldn't
> fail:
> {code:scala}
> InSet(Literal("a"), Set("a", "b").map(UTF8String.fromString)).sql
> {code}
> but it fails with the exception:
> {code}
> Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a
> java.lang.RuntimeException: Unsupported literal type class
> org.apache.spark.unsafe.types.UTF8String a
> at
> org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:88)
> at
> org.apache.spark.sql.catalyst.expressions.InSet.$anonfun$sql$2(predicates.scala:522)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-31564) Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
Kent Yao created SPARK-31564: Summary: Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens Key: SPARK-31564 URL: https://issues.apache.org/jira/browse/SPARK-31564 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.0.0, 3.1.0 Reporter: Kent Yao the org.apache.spark.sql.execution.ui.AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens is flaky for just checking the html not containing 1970. I will add a ticket to check and fix that. In the specific failure https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121799/testReport, it failed because ... ... contained 1970 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31564) Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
[ https://issues.apache.org/jira/browse/SPARK-31564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-31564: - Affects Version/s: 2.4.5 > Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page > when event reordering happens > --- > > Key: SPARK-31564 > URL: https://issues.apache.org/jira/browse/SPARK-31564 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.5, 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Major > > the > org.apache.spark.sql.execution.ui.AllExecutionsPageSuite.SPARK-27019:correctly > display SQL page when event reordering happens is flaky for just checking > the html not containing 1970. I will add a ticket to check and fix that. > In the specific failure > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121799/testReport, > it failed because > ... > > ... > contained 1970 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31564) Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
[ https://issues.apache.org/jira/browse/SPARK-31564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-31564: - Affects Version/s: (was: 2.4.5) > Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page > when event reordering happens > --- > > Key: SPARK-31564 > URL: https://issues.apache.org/jira/browse/SPARK-31564 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Priority: Major > > the > org.apache.spark.sql.execution.ui.AllExecutionsPageSuite.SPARK-27019:correctly > display SQL page when event reordering happens is flaky for just checking > the html not containing 1970. I will add a ticket to check and fix that. > In the specific failure > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121799/testReport, > it failed because > ... > > ... > contained 1970 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31563) Failure of InSet.sql for UTF8String collection
[
https://issues.apache.org/jira/browse/SPARK-31563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-31563.
---
Fix Version/s: 3.0.0
2.4.6
Resolution: Fixed
Issue resolved by pull request 28343
[https://github.com/apache/spark/pull/28343]
> Failure of InSet.sql for UTF8String collection
> --
>
> Key: SPARK-31563
> URL: https://issues.apache.org/jira/browse/SPARK-31563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.5, 3.0.0, 3.1.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Major
> Fix For: 2.4.6, 3.0.0
>
>
> The InSet expression works on collections of internal Catalyst's types. We
> can see this in the optimization when In is replaced by InSet, and In's
> collection is evaluated to internal Catalyst's values:
> [https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L254]
> {code:scala}
> if (newList.length > SQLConf.get.optimizerInSetConversionThreshold) {
> val hSet = newList.map(e => e.eval(EmptyRow))
> InSet(v, HashSet() ++ hSet)
> }
> {code}
> The code existed before the optimization
> https://github.com/apache/spark/pull/25754 that made another wrong assumption
> about collection types.
> If InSet accepts only internal Catalyst's types, the following code shouldn't
> fail:
> {code:scala}
> InSet(Literal("a"), Set("a", "b").map(UTF8String.fromString)).sql
> {code}
> but it fails with the exception:
> {code}
> Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a
> java.lang.RuntimeException: Unsupported literal type class
> org.apache.spark.unsafe.types.UTF8String a
> at
> org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:88)
> at
> org.apache.spark.sql.catalyst.expressions.InSet.$anonfun$sql$2(predicates.scala:522)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31563) Failure of InSet.sql for UTF8String collection
[
https://issues.apache.org/jira/browse/SPARK-31563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-31563:
-
Assignee: Maxim Gekk
> Failure of InSet.sql for UTF8String collection
> --
>
> Key: SPARK-31563
> URL: https://issues.apache.org/jira/browse/SPARK-31563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.5, 3.0.0, 3.1.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Major
>
> The InSet expression works on collections of internal Catalyst's types. We
> can see this in the optimization when In is replaced by InSet, and In's
> collection is evaluated to internal Catalyst's values:
> [https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L254]
> {code:scala}
> if (newList.length > SQLConf.get.optimizerInSetConversionThreshold) {
> val hSet = newList.map(e => e.eval(EmptyRow))
> InSet(v, HashSet() ++ hSet)
> }
> {code}
> The code existed before the optimization
> https://github.com/apache/spark/pull/25754 that made another wrong assumption
> about collection types.
> If InSet accepts only internal Catalyst's types, the following code shouldn't
> fail:
> {code:scala}
> InSet(Literal("a"), Set("a", "b").map(UTF8String.fromString)).sql
> {code}
> but it fails with the exception:
> {code}
> Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a
> java.lang.RuntimeException: Unsupported literal type class
> org.apache.spark.unsafe.types.UTF8String a
> at
> org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:88)
> at
> org.apache.spark.sql.catalyst.expressions.InSet.$anonfun$sql$2(predicates.scala:522)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31528) Remove millennium, century, decade from trunc/date_trunc fucntions
[ https://issues.apache.org/jira/browse/SPARK-31528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31528. --- Fix Version/s: 3.0.0 Assignee: Kent Yao Resolution: Fixed This is resolved via the followings. - master: https://github.com/apache/spark/pull/28313 - branch-3.0: https://github.com/apache/spark/pull/28340 > Remove millennium, century, decade from trunc/date_trunc fucntions > --- > > Key: SPARK-31528 > URL: https://issues.apache.org/jira/browse/SPARK-31528 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 3.0.0 > > > Same as #SPARK-31507, millennium, century, and decade are not commonly used > in most modern platforms. > for example > Negative: > https://docs.snowflake.com/en/sql-reference/functions-date-time.html#supported-date-and-time-parts > https://prestodb.io/docs/current/functions/datetime.html#date_trunc > https://teradata.github.io/presto/docs/148t/functions/datetime.html#date_trunc > https://www.oracletutorial.com/oracle-date-functions/oracle-trunc/ > Positive: > https://docs.aws.amazon.com/redshift/latest/dg/r_Dateparts_for_datetime_functions.html > https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31540: -- Target Version/s: (was: 2.4.6) > Backport SPARK-27981 Remove `Illegal reflective access` warning for > `java.nio.Bits.unaligned()` in JDK9+ > -- > > Key: SPARK-31540 > URL: https://issues.apache.org/jira/browse/SPARK-31540 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.4.6 >Reporter: Holden Karau >Priority: Major > > SPARK-27981 Remove `Illegal reflective access` warning for > `java.nio.Bits.unaligned()` in JDK9+ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-31540. - > Backport SPARK-27981 Remove `Illegal reflective access` warning for > `java.nio.Bits.unaligned()` in JDK9+ > -- > > Key: SPARK-31540 > URL: https://issues.apache.org/jira/browse/SPARK-31540 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.4.6 >Reporter: Holden Karau >Priority: Major > > SPARK-27981 Remove `Illegal reflective access` warning for > `java.nio.Bits.unaligned()` in JDK9+ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31540. --- Resolution: Won't Do Since there is no consensus on `Improvement` JIRA, I'll close this as `Won't Do` according to our basic backporting principle. > Backport SPARK-27981 Remove `Illegal reflective access` warning for > `java.nio.Bits.unaligned()` in JDK9+ > -- > > Key: SPARK-31540 > URL: https://issues.apache.org/jira/browse/SPARK-31540 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.4.6 >Reporter: Holden Karau >Priority: Major > > SPARK-27981 Remove `Illegal reflective access` warning for > `java.nio.Bits.unaligned()` in JDK9+ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31564) Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
[ https://issues.apache.org/jira/browse/SPARK-31564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31564. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28344 [https://github.com/apache/spark/pull/28344] > Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page > when event reordering happens > --- > > Key: SPARK-31564 > URL: https://issues.apache.org/jira/browse/SPARK-31564 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Fix For: 3.0.0 > > > the > org.apache.spark.sql.execution.ui.AllExecutionsPageSuite.SPARK-27019:correctly > display SQL page when event reordering happens is flaky for just checking > the html not containing 1970. I will add a ticket to check and fix that. > In the specific failure > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121799/testReport, > it failed because > ... > > ... > contained 1970 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31564) Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
[ https://issues.apache.org/jira/browse/SPARK-31564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31564: - Assignee: Kent Yao > Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page > when event reordering happens > --- > > Key: SPARK-31564 > URL: https://issues.apache.org/jira/browse/SPARK-31564 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > > the > org.apache.spark.sql.execution.ui.AllExecutionsPageSuite.SPARK-27019:correctly > display SQL page when event reordering happens is flaky for just checking > the html not containing 1970. I will add a ticket to check and fix that. > In the specific failure > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121799/testReport, > it failed because > ... > > ... > contained 1970 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31557) Legacy parser incorrectly interprets pre-Gregorian dates
[
https://issues.apache.org/jira/browse/SPARK-31557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092318#comment-17092318
]
Bruce Robbins commented on SPARK-31557:
---
In case Jiras are no longer getting updated, here's a PR:
https://github.com/apache/spark/pull/28345
> Legacy parser incorrectly interprets pre-Gregorian dates
>
>
> Key: SPARK-31557
> URL: https://issues.apache.org/jira/browse/SPARK-31557
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Bruce Robbins
>Priority: Major
>
> With CSV:
> {noformat}
> scala> sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
> res0: org.apache.spark.sql.DataFrame = [key: string, value: string]
> scala> val seq = Seq("0002-01-01", "1000-01-01", "1500-01-01",
> "1800-01-01").map(x => s"$x,$x")
> seq: Seq[String] = List(0002-01-01,0002-01-01, 1000-01-01,1000-01-01,
> 1500-01-01,1500-01-01, 1800-01-01,1800-01-01)
> scala> val ds = seq.toDF("value").as[String]
> ds: org.apache.spark.sql.Dataset[String] = [value: string]
> scala> spark.read.schema("expected STRING, actual DATE").csv(ds).show
> +--+--+
> | expected|actual|
> +--+--+
> |0002-01-01|0001-12-30|
> |1000-01-01|1000-01-06|
> |1500-01-01|1500-01-10|
> |1800-01-01|1800-01-01|
> +--+--+
> scala>
> {noformat}
> Similarly, with JSON:
> {noformat}
> scala> sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
> res0: org.apache.spark.sql.DataFrame = [key: string, value: string]
> scala> val seq = Seq("0002-01-01", "1000-01-01", "1500-01-01",
> "1800-01-01").map { x =>
> s"""{"expected": "$x", "actual": "$x"}"""
> }
> | | seq: Seq[String] = List({"expected": "0002-01-01", "actual":
> "0002-01-01"}, {"expected": "1000-01-01", "actual": "1000-01-01"},
> {"expected": "1500-01-01", "actual": "1500-01-01"}, {"expected":
> "1800-01-01", "actual": "1800-01-01"})
> scala>
> scala> val ds = seq.toDF("value").as[String]
> ds: org.apache.spark.sql.Dataset[String] = [value: string]
> scala> spark.read.schema("expected STRING, actual DATE").json(ds).show
> +--+--+
> | expected|actual|
> +--+--+
> |0002-01-01|0001-12-30|
> |1000-01-01|1000-01-06|
> |1500-01-01|1500-01-10|
> |1800-01-01|1800-01-01|
> +--+--+
> scala>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31364) Benchmark Nested Parquet Predicate Pushdown
[ https://issues.apache.org/jira/browse/SPARK-31364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31364: - Assignee: Jian TANG > Benchmark Nested Parquet Predicate Pushdown > --- > > Key: SPARK-31364 > URL: https://issues.apache.org/jira/browse/SPARK-31364 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: DB Tsai >Assignee: Jian TANG >Priority: Major > Fix For: 3.0.0 > > > We would like to benchmark best and worst scenarios such as no record matches > the predicate, and how much extra overhead is added. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31558) Code cleanup in spark-sql-viz.js
[ https://issues.apache.org/jira/browse/SPARK-31558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-31558. Resolution: Fixed This issue is resolved in https://github.com/apache/spark/pull/28333 > Code cleanup in spark-sql-viz.js > > > Key: SPARK-31558 > URL: https://issues.apache.org/jira/browse/SPARK-31558 > Project: Spark > Issue Type: Task > Components: Web UI >Affects Versions: 3.0.0, 3.1.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > > 1. Remove console.log(), which seems unnecessary in release. > 2. Replace the double equals to triple equals > 3. Reuse jquery selector. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31552) Fix potential ClassCastException in ScalaReflection arrayClassFor
[
https://issues.apache.org/jira/browse/SPARK-31552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-31552:
--
Fix Version/s: 2.4.6
> Fix potential ClassCastException in ScalaReflection arrayClassFor
> -
>
> Key: SPARK-31552
> URL: https://issues.apache.org/jira/browse/SPARK-31552
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.5, 3.0.0, 3.1.0
>Reporter: Kent Yao
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.4.6, 3.0.0
>
>
> arrayClassFor and dataTypeFor in ScalaReflection call each other circularly,
> the cases in dataTypeFor are not fully handled in arrayClassFor
> For example:
> {code:java}
> scala> import scala.reflect.runtime.universe.TypeTag
> scala> import org.apache.spark.sql._
> scala> import org.apache.spark.sql.catalyst.encoders._
> scala> import org.apache.spark.sql.types._
> scala> implicit def newArrayEncoder[T <: Array[_] : TypeTag]: Encoder[T] =
> ExpressionEncoder()
> newArrayEncoder: [T <: Array[_]](implicit evidence$1:
> reflect.runtime.universe.TypeTag[T])org.apache.spark.sql.Encoder[T]
> scala> val decOne = Decimal(1, 38, 18)
> decOne: org.apache.spark.sql.types.Decimal = 1E-18
> scala> val decTwo = Decimal(2, 38, 18)
> decTwo: org.apache.spark.sql.types.Decimal = 2E-18
> scala> val decSpark = Array(decOne, decTwo)
> decSpark: Array[org.apache.spark.sql.types.Decimal] = Array(1E-18, 2E-18)
> scala> Seq(decSpark).toDF()
> java.lang.ClassCastException: org.apache.spark.sql.types.DecimalType cannot
> be cast to org.apache.spark.sql.types.ObjectType
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$arrayClassFor$1(ScalaReflection.scala:131)
> at
> scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects(ScalaReflection.scala:879)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects$(ScalaReflection.scala:878)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:49)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.arrayClassFor(ScalaReflection.scala:120)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$dataTypeFor$1(ScalaReflection.scala:105)
> at
> scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects(ScalaReflection.scala:879)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects$(ScalaReflection.scala:878)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:49)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.dataTypeFor(ScalaReflection.scala:88)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$serializerForType$1(ScalaReflection.scala:399)
> at
> scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects(ScalaReflection.scala:879)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects$(ScalaReflection.scala:878)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:49)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.serializerForType(ScalaReflection.scala:393)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:57)
> at newArrayEncoder(:57)
> ... 53 elided
> scala>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
