[jira] [Updated] (SPARK-38162) Optimize one row plan in normal and AQE Optimizer

2022-02-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Summary: Optimize one row plan in normal and AQE Optimizer (was: Optimize one max row plan in normal

[jira] [Updated] (SPARK-38162) Optimize one max row plan in normal and AQE Optimizer

2022-02-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Description: Optimize the plan if its max row is equal to or less than 1 in these cases: * if sort

[jira] [Updated] (SPARK-38162) Optimize one max row plan in normal and AQE Optimizer

2022-02-09 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Summary: Optimize one max row plan in normal and AQE Optimizer (was: Remove distinct in aggregate if

[jira] [Assigned] (SPARK-38157) Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38157: --- Assignee: Xinyi Yu > Fix >

[jira] [Resolved] (SPARK-38157) Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38157. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35471

[jira] [Commented] (SPARK-38161) when clean data hope to spilt one dataframe or dataset to two dataframe

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489973#comment-17489973 ] Hyukjin Kwon commented on SPARK-38161: -- no I meant the API in Scala itself: {code} scala> Seq(1,

[jira] [Commented] (SPARK-38161) when clean data hope to spilt one dataframe or dataset to two dataframe

2022-02-09 Thread gaokui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489971#comment-17489971 ] gaokui commented on SPARK-38161: do you mean that is 'wirte.mode('parquet').patitionby('col')' or

[jira] [Resolved] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Dooyoung Hwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dooyoung Hwang resolved SPARK-38168. Resolution: Won't Fix > LikeSimplification handles escape character >

[jira] [Commented] (SPARK-38165) private classes fail at runtime in scala 2.12.13+

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489965#comment-17489965 ] Hyukjin Kwon commented on SPARK-38165: -- Okay, so I guess the problem is here:

[jira] [Updated] (SPARK-38165) private classes fail at runtime in scala 2.12.13+

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38165: - Priority: Minor (was: Major) > private classes fail at runtime in scala 2.12.13+ >

[jira] [Commented] (SPARK-38161) when clean data hope to spilt one dataframe or dataset to two dataframe

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489956#comment-17489956 ] Hyukjin Kwon commented on SPARK-38161: -- I guess you want an API like partition. Even if we

[jira] [Updated] (SPARK-38161) when clean data hope to spilt one dataframe or dataset to two dataframe

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38161: - Component/s: SQL (was: Block Manager) > when clean data hope to spilt one

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-02-09 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Assigned] (SPARK-38170) Fix //sql/hive-thriftserver:org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextInHttpSuite-hive-2.3__hadoop-2.7 in ANSI

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38170: Assignee: Apache Spark > Fix >

[jira] [Assigned] (SPARK-38170) Fix //sql/hive-thriftserver:org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextInHttpSuite-hive-2.3__hadoop-2.7 in ANSI

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38170: Assignee: (was: Apache Spark) > Fix >

[jira] [Commented] (SPARK-38170) Fix //sql/hive-thriftserver:org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextInHttpSuite-hive-2.3__hadoop-2.7 in ANSI

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489952#comment-17489952 ] Apache Spark commented on SPARK-38170: -- User 'amaliujia' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-38146: -- Assignee: Bruce Robbins > UDAF fails to aggregate TIMESTAMP_NTZ column >

[jira] [Resolved] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38146. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35470

[jira] [Created] (SPARK-38170) Fix //sql/hive-thriftserver:org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextInHttpSuite-hive-2.3__hadoop-2.7 in ANSI

2022-02-09 Thread Rui Wang (Jira)
Rui Wang created SPARK-38170: Summary: Fix //sql/hive-thriftserver:org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextInHttpSuite-hive-2.3__hadoop-2.7 in ANSI Key: SPARK-38170 URL:

[jira] [Commented] (SPARK-38165) private classes fail at runtime in scala 2.12.13+

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489945#comment-17489945 ] Hyukjin Kwon commented on SPARK-38165: -- Interesting. Spark already uses Scala 2.12.15 even in Spark

[jira] [Resolved] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38163. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35467

[jira] [Assigned] (SPARK-38113) Use error classes in the execution errors of pivoting

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38113: Assignee: (was: Apache Spark) > Use error classes in the execution errors of

[jira] [Assigned] (SPARK-38113) Use error classes in the execution errors of pivoting

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38113: Assignee: Apache Spark > Use error classes in the execution errors of pivoting >

[jira] [Commented] (SPARK-38113) Use error classes in the execution errors of pivoting

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489920#comment-17489920 ] Apache Spark commented on SPARK-38113: -- User 'ivoson' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37406) Inline type hints for python/pyspark/ml/fpm.py

2022-02-09 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37406. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-38157) Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38157: Assignee: Apache Spark > Fix >

[jira] [Commented] (SPARK-38157) Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489910#comment-17489910 ] Apache Spark commented on SPARK-38157: -- User 'anchovYu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38157) Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38157: Assignee: (was: Apache Spark) > Fix >

[jira] [Commented] (SPARK-38157) Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489909#comment-17489909 ] Apache Spark commented on SPARK-38157: -- User 'anchovYu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37406) Inline type hints for python/pyspark/ml/fpm.py

2022-02-09 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37406: -- Assignee: Maciej Szymkiewicz > Inline type hints for

[jira] [Resolved] (SPARK-38164) New SQL function: try_subtract and try_multiply

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38164. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35461

[jira] [Commented] (SPARK-38139) ml.recommendation.ALS doctests failures

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489884#comment-17489884 ] Hyukjin Kwon commented on SPARK-38139: -- yup agree > ml.recommendation.ALS doctests failures >

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489880#comment-17489880 ] Dongjoon Hyun commented on SPARK-37814: --- [~kabhwan]. Yes, it could be. Let's wait and see the way

[jira] [Assigned] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38146: Assignee: (was: Apache Spark) > UDAF fails to aggregate TIMESTAMP_NTZ column >

[jira] [Assigned] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38146: Assignee: Apache Spark > UDAF fails to aggregate TIMESTAMP_NTZ column >

[jira] [Commented] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489878#comment-17489878 ] Apache Spark commented on SPARK-38146: -- User 'bersprockets' has created a pull request for this

[jira] [Updated] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38146: -- Summary: UDAF fails to aggregate TIMESTAMP_NTZ column (was: UDAF fails with unsafe row

[jira] [Comment Edited] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489857#comment-17489857 ] Jungtaek Lim edited comment on SPARK-37814 at 2/9/22, 11:03 PM: If we

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489857#comment-17489857 ] Jungtaek Lim commented on SPARK-37814: -- If we are open to release for maintenance for versions

[jira] [Updated] (SPARK-38146) UDAF fails with unsafe row buffer containing a TIMESTAMP_NTZ column

2022-02-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38146: -- Summary: UDAF fails with unsafe row buffer containing a TIMESTAMP_NTZ column (was: UDAF

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489821#comment-17489821 ] Dongjoon Hyun commented on SPARK-37814: --- Thank you for sharing, [~ste...@apache.org]. And, thank

[jira] [Updated] (SPARK-38151) Handle `Pacific/Kanton` in DateTimeUtilsSuite

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Description: This issue aims to fix the flaky UT failures due to

[jira] [Updated] (SPARK-38151) Handle `Pacific/Kanton` in DateTimeUtilsSuite

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Summary: Handle `Pacific/Kanton` in DateTimeUtilsSuite (was: Flaky Test:

[jira] [Resolved] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38151. --- Fix Version/s: 3.3.0 3.2.2 3.1.3 Resolution:

[jira] [Assigned] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38151: - Assignee: Dongjoon Hyun > Flaky Test: DateTimeUtilsSuite.`daysToMicros and

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Sujit Biswas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489793#comment-17489793 ] Sujit Biswas commented on SPARK-38061: -- [~abhinavofficial] fyi, I was able to build the

[jira] [Assigned] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38155: Assignee: (was: Apache Spark) > Disallow distinct aggregate in lateral subqueries

[jira] [Commented] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489747#comment-17489747 ] Apache Spark commented on SPARK-38155: -- User 'allisonwang-db' has created a pull request for this

[jira] [Assigned] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38155: Assignee: Apache Spark > Disallow distinct aggregate in lateral subqueries with

[jira] [Updated] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-38155: - Description: Block lateral subqueries in CheckAnalysis that contain DISTINCT aggregate and

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Stephen L. De Rudder (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489742#comment-17489742 ] Stephen L. De Rudder commented on SPARK-37814: -- Thank you for the responses, [~dongjoon]

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Affects Version/s: 3.1.3 > Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays` >

[jira] [Commented] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489732#comment-17489732 ] Apache Spark commented on SPARK-38151: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38151: Assignee: Apache Spark > Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

[jira] [Commented] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489731#comment-17489731 ] Apache Spark commented on SPARK-38151: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38151: Assignee: (was: Apache Spark) > Flaky Test: DateTimeUtilsSuite.`daysToMicros and

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Description: **MASTER** -

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Affects Version/s: 3.2.2 > Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays` >

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Description: **MASTER** -

[jira] [Created] (SPARK-38169) Use OffHeap memory if configured in vectorized DeltaByteArray reader

2022-02-09 Thread Parth Chandra (Jira)
Parth Chandra created SPARK-38169: - Summary: Use OffHeap memory if configured in vectorized DeltaByteArray reader Key: SPARK-38169 URL: https://issues.apache.org/jira/browse/SPARK-38169 Project:

[jira] [Updated] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-38155: - Description: Block lateral subqueries in CheckAnalysis that contains DISTINCT aggregate and

[jira] [Comment Edited] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Sujit Biswas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489711#comment-17489711 ] Sujit Biswas edited comment on SPARK-38061 at 2/9/22, 5:56 PM: ---

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Sujit Biswas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489711#comment-17489711 ] Sujit Biswas commented on SPARK-38061: -- [~abhinavofficial] there are multiple vulnerabilities, as

[jira] [Updated] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38056: -- Fix Version/s: 3.1.3 3.2.2 > Structured streaming not working in history

[jira] [Updated] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38056: -- Target Version/s: (was: 3.1.2, 3.2.0) > Structured streaming not working in history server

[jira] [Updated] (SPARK-38120) HiveExternalCatalog.listPartitions is failing when partition column name is upper case and dot in partition value

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38120: -- Fix Version/s: 3.1.3 > HiveExternalCatalog.listPartitions is failing when partition column

[jira] [Updated] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-38166: -- Description: We're seeing duplicates after running the following {code:java} def

[jira] [Commented] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489670#comment-17489670 ] Apache Spark commented on SPARK-38163: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489668#comment-17489668 ] Apache Spark commented on SPARK-38163: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38163: Assignee: Apache Spark (was: Max Gekk) > Preserve the error class of

[jira] [Assigned] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38163: Assignee: Max Gekk (was: Apache Spark) > Preserve the error class of

[jira] [Updated] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38163: - Description: When the cause exception is `AnalysisException` at

[jira] [Assigned] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38168: Assignee: Apache Spark > LikeSimplification handles escape character >

[jira] [Commented] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489611#comment-17489611 ] Apache Spark commented on SPARK-38168: -- User 'Dooyoung-Hwang' has created a pull request for this

[jira] [Commented] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489612#comment-17489612 ] Apache Spark commented on SPARK-38168: -- User 'Dooyoung-Hwang' has created a pull request for this

[jira] [Assigned] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38168: Assignee: (was: Apache Spark) > LikeSimplification handles escape character >

[jira] [Resolved] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38061. -- Resolution: Incomplete > security scan issue with htrace-core4-4.1.0-incubating >

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489608#comment-17489608 ] Hyukjin Kwon commented on SPARK-38061: -- Again, this doesn't directly affect Spark since Spark

[jira] [Commented] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489591#comment-17489591 ] Apache Spark commented on SPARK-38056: -- User 'kuwii' has created a pull request for this issue:

[jira] [Commented] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489588#comment-17489588 ] Apache Spark commented on SPARK-38056: -- User 'kuwii' has created a pull request for this issue:

[jira] [Commented] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489589#comment-17489589 ] Apache Spark commented on SPARK-38056: -- User 'kuwii' has created a pull request for this issue:

[jira] [Created] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Dooyoung Hwang (Jira)
Dooyoung Hwang created SPARK-38168: -- Summary: LikeSimplification handles escape character Key: SPARK-38168 URL: https://issues.apache.org/jira/browse/SPARK-38168 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-37969) Hive Serde insert should check schema before execution

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37969: --- Assignee: angerszhu > Hive Serde insert should check schema before execution >

[jira] [Resolved] (SPARK-37969) Hive Serde insert should check schema before execution

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37969. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35258

[jira] [Commented] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489579#comment-17489579 ] Willi Raschkowski commented on SPARK-38166: --- Linking SPARK-23207 (which is closed but looks

[jira] [Assigned] (SPARK-37585) DSV2 InputMetrics are not getting update in corner case

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37585: --- Assignee: Sandeep Katta > DSV2 InputMetrics are not getting update in corner case >

[jira] [Resolved] (SPARK-37585) DSV2 InputMetrics are not getting update in corner case

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37585. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35432

[jira] [Resolved] (SPARK-37652) Support optimize skewed join through union

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37652. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34908

[jira] [Updated] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-09 Thread Marnix van den Broek (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marnix van den Broek updated SPARK-38167: - Description: hi all, When reading CSV files with Spark, I ran into a parsing

[jira] [Updated] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-09 Thread Marnix van den Broek (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marnix van den Broek updated SPARK-38167: - Description: hi all, When reading CSV files with Spark, I ran into a parsing

[jira] [Created] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-09 Thread Marnix van den Broek (Jira)
Marnix van den Broek created SPARK-38167: Summary: CSV parsing error when using escape='"' Key: SPARK-38167 URL: https://issues.apache.org/jira/browse/SPARK-38167 Project: Spark

[jira] [Commented] (SPARK-34511) Current Security vulnerabilities in spark libraries

2022-02-09 Thread Abhinav Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489544#comment-17489544 ] Abhinav Kumar commented on SPARK-34511: --- Updating to Spark 3.2.1 does solve most of the issues.

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Abhinav Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489537#comment-17489537 ] Abhinav Kumar commented on SPARK-38061: --- [~hyukjin.kwon] [~sujitbiswas] Are we agreeing to track

[jira] [Updated] (SPARK-36145) Remove Python 3.6 support in codebase and CI

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36145: - Labels: release-notes (was: ) > Remove Python 3.6 support in codebase and CI >

  1   2   >