[jira] [Assigned] (SPARK-35261) Support static invoke for stateless UDF

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35261: Assignee: Apache Spark > Support static invoke for stateless UDF > --

[jira] [Commented] (SPARK-35261) Support static invoke for stateless UDF

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337133#comment-17337133 ] Apache Spark commented on SPARK-35261: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-35261) Support static invoke for stateless UDF

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35261: Assignee: (was: Apache Spark) > Support static invoke for stateless UDF > ---

[jira] [Assigned] (SPARK-35261) Support static invoke for stateless UDF

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35261: Assignee: Apache Spark > Support static invoke for stateless UDF > --

[jira] [Commented] (SPARK-35280) Promote KubernetesUtils to DeveloperApi

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337129#comment-17337129 ] Apache Spark commented on SPARK-35280: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35280) Promote KubernetesUtils to DeveloperApi

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337130#comment-17337130 ] Apache Spark commented on SPARK-35280: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-35280) Promote KubernetesUtils to DeveloperApi

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35280: Assignee: (was: Apache Spark) > Promote KubernetesUtils to DeveloperApi > ---

[jira] [Assigned] (SPARK-35280) Promote KubernetesUtils to DeveloperApi

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35280: Assignee: Apache Spark > Promote KubernetesUtils to DeveloperApi > --

[jira] [Commented] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337128#comment-17337128 ] Apache Spark commented on SPARK-35273: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35273: Assignee: Apache Spark > CombineFilters support non-deterministic expressions > -

[jira] [Assigned] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35273: Assignee: (was: Apache Spark) > CombineFilters support non-deterministic expressions

[jira] [Created] (SPARK-35280) Promote KubernetesUtils to DeveloperApi

2021-04-29 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-35280: - Summary: Promote KubernetesUtils to DeveloperApi Key: SPARK-35280 URL: https://issues.apache.org/jira/browse/SPARK-35280 Project: Spark Issue Type: Improve

[jira] [Assigned] (SPARK-35271) After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35271: Assignee: Weichen Xu (was: Apache Spark) > After CrossValidator/TrainValidationSplit fit

[jira] [Commented] (SPARK-35271) After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337116#comment-17337116 ] Apache Spark commented on SPARK-35271: -- User 'WeichenXu123' has created a pull requ

[jira] [Assigned] (SPARK-35271) After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35271: Assignee: Apache Spark (was: Weichen Xu) > After CrossValidator/TrainValidationSplit fit

[jira] [Updated] (SPARK-35270) Remove the use of guava in order to upgrade guava version to 27

2021-04-29 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Wang updated SPARK-35270: -- Summary: Remove the use of guava in order to upgrade guava version to 27 (was: Remove the use of guav

[jira] [Assigned] (SPARK-35111) Cast string to year-month interval

2021-04-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-35111: Assignee: angerszhu > Cast string to year-month interval > -- > >

[jira] [Resolved] (SPARK-35111) Cast string to year-month interval

2021-04-29 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-35111. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32266 [https://github.com

[jira] [Assigned] (SPARK-35277) Upgrade snappy to 1.1.8.4

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35277: - Assignee: William Hyun > Upgrade snappy to 1.1.8.4 > - > >

[jira] [Resolved] (SPARK-35277) Upgrade snappy to 1.1.8.4

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35277. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32402 [https://

[jira] [Created] (SPARK-35279) _SUCCESS file not written when using partitionOverwriteMode=dynamic

2021-04-29 Thread Goran Kljajic (Jira)
Goran Kljajic created SPARK-35279: - Summary: _SUCCESS file not written when using partitionOverwriteMode=dynamic Key: SPARK-35279 URL: https://issues.apache.org/jira/browse/SPARK-35279 Project: Spark

[jira] [Assigned] (SPARK-35278) Invoke should find the method with correct number of parameters

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35278: Assignee: L. C. Hsieh (was: Apache Spark) > Invoke should find the method with correct n

[jira] [Commented] (SPARK-35278) Invoke should find the method with correct number of parameters

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337046#comment-17337046 ] Apache Spark commented on SPARK-35278: -- User 'viirya' has created a pull request fo

[jira] [Assigned] (SPARK-35278) Invoke should find the method with correct number of parameters

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35278: Assignee: Apache Spark (was: L. C. Hsieh) > Invoke should find the method with correct n

[jira] [Created] (SPARK-35278) Invoke should find the method with correct number of parameters

2021-04-29 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-35278: --- Summary: Invoke should find the method with correct number of parameters Key: SPARK-35278 URL: https://issues.apache.org/jira/browse/SPARK-35278 Project: Spark

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet") spark.sq

[jira] [Assigned] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35255: Assignee: Zhu, Lipeng > Automated formatting for Scala Code for Blank Lines. > --

[jira] [Resolved] (SPARK-35255) Automated formatting for Scala Code for Blank Lines.

2021-04-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35255. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32383 [https://gi

[jira] [Resolved] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35184. -- Resolution: Incomplete > Filtering a dataframe after groupBy and user-define-aggregate-functio

[jira] [Commented] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337027#comment-17337027 ] Hyukjin Kwon commented on SPARK-35184: -- [~xiaoking] Spark 2.4's last release is out

[jira] [Commented] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

2021-04-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337023#comment-17337023 ] Hyukjin Kwon commented on SPARK-26224: -- There's a discussion going on in the dev ma

[jira] [Updated] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-29 Thread Xiao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Jin updated SPARK-35184: - Description: I found some strange error when I'm coding Pyspark UDAF. After I call groupBy function and

[jira] [Reopened] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-29 Thread Xiao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Jin reopened SPARK-35184: -- I think in the PushDownPredicate in branch-2.4 at the case of Filter(condition, aggregate: Aggregate), wr

[jira] [Assigned] (SPARK-35277) Upgrade snappy to 1.1.8.4

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35277: Assignee: Apache Spark > Upgrade snappy to 1.1.8.4 > - > >

[jira] [Commented] (SPARK-35277) Upgrade snappy to 1.1.8.4

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336620#comment-17336620 ] Apache Spark commented on SPARK-35277: -- User 'williamhyun' has created a pull reque

[jira] [Assigned] (SPARK-35277) Upgrade snappy to 1.1.8.4

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35277: Assignee: (was: Apache Spark) > Upgrade snappy to 1.1.8.4 > -

[jira] [Created] (SPARK-35277) Upgrade snappy to 1.1.8.4

2021-04-29 Thread William Hyun (Jira)
William Hyun created SPARK-35277: Summary: Upgrade snappy to 1.1.8.4 Key: SPARK-35277 URL: https://issues.apache.org/jira/browse/SPARK-35277 Project: Spark Issue Type: Improvement C

[jira] [Commented] (SPARK-35054) Getting Critical Vulnerability CVE-2021-20231 on spark 3.0.0 branch

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335814#comment-17335814 ] Dongjoon Hyun commented on SPARK-35054: --- Hi, [~jainshasha] . Apache Spark doesn't

[jira] [Commented] (SPARK-34731) ConcurrentModificationException in EventLoggingListener when redacting properties

2021-04-29 Thread Rui Balau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335711#comment-17335711 ] Rui Balau commented on SPARK-34731: --- I'd just like to add, while either 3.1.2. or 3.2.

[jira] [Resolved] (SPARK-35047) Allow Json datasources to write non-ascii characters as codepoints

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35047. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32147 [https://

[jira] [Resolved] (SPARK-35268) Upgrade GenJavadoc to 0.17

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35268. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32392 [https://

[jira] [Resolved] (SPARK-35009) Avoid creating multiple Monitor threads for reused python workers for the same TaskContext

2021-04-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35009. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32169

[jira] [Resolved] (SPARK-35254) Upgrade SBT to 1.5.1

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35254. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32382 [https://

[jira] [Assigned] (SPARK-35254) Upgrade SBT to 1.5.1

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35254: - Assignee: Zhu, Lipeng > Upgrade SBT to 1.5.1 > > >

[jira] [Resolved] (SPARK-35269) Update commons-lang3 to 3.12

2021-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35269. --- Fix Version/s: 3.2.0 Assignee: Yang Jie Resolution: Fixed This is resolved v

[jira] [Assigned] (SPARK-35275) Add checksum for shuffle blocks

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35275: Assignee: (was: Apache Spark) > Add checksum for shuffle blocks > ---

[jira] [Assigned] (SPARK-35275) Add checksum for shuffle blocks

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35275: Assignee: Apache Spark > Add checksum for shuffle blocks > --

[jira] [Commented] (SPARK-35275) Add checksum for shuffle blocks

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335654#comment-17335654 ] Apache Spark commented on SPARK-35275: -- User 'Ngone51' has created a pull request f

[jira] [Commented] (SPARK-35276) Write checksum files for shuffle

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335652#comment-17335652 ] Apache Spark commented on SPARK-35276: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-35276) Write checksum files for shuffle

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35276: Assignee: (was: Apache Spark) > Write checksum files for shuffle > -

[jira] [Commented] (SPARK-35276) Write checksum files for shuffle

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335649#comment-17335649 ] Apache Spark commented on SPARK-35276: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-35276) Write checksum files for shuffle

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35276: Assignee: Apache Spark > Write checksum files for shuffle >

[jira] [Assigned] (SPARK-29771) Limit executor max failures before failing the application

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29771: Assignee: Apache Spark > Limit executor max failures before failing the application > ---

[jira] [Assigned] (SPARK-29771) Limit executor max failures before failing the application

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29771: Assignee: (was: Apache Spark) > Limit executor max failures before failing the applic

[jira] [Created] (SPARK-35276) Write checksum files for shuffle

2021-04-29 Thread wuyi (Jira)
wuyi created SPARK-35276: Summary: Write checksum files for shuffle Key: SPARK-35276 URL: https://issues.apache.org/jira/browse/SPARK-35276 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-35275) Add checksum for shuffle blocks

2021-04-29 Thread wuyi (Jira)
wuyi created SPARK-35275: Summary: Add checksum for shuffle blocks Key: SPARK-35275 URL: https://issues.apache.org/jira/browse/SPARK-35275 Project: Spark Issue Type: New Feature Components:

[jira] [Commented] (SPARK-35184) Filtering a dataframe after groupBy and user-define-aggregate-function in Pyspark will cause java.lang.UnsupportedOperationException

2021-04-29 Thread Xiao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335636#comment-17335636 ] Xiao Jin commented on SPARK-35184: -- What I guess above is wrong. In branch 2.4, I thin

[jira] [Assigned] (SPARK-35234) Reserve format of the stage failureMessage

2021-04-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35234: -- Assignee: wuyi > Reserve format of the stage failureMessage > ---

[jira] [Resolved] (SPARK-35234) Reserve format of the stage failureMessage

2021-04-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35234. Fix Version/s: 3.2.0 Resolution: Fixed > Reserve format of the stage failur

[jira] [Updated] (SPARK-35274) old hive table's all columns are read when column pruning applies in spark3.0

2021-04-29 Thread xiaoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaoli updated SPARK-35274: --- Description: I asked this question [before|https://issues.apache.org/jira/browse/SPARK-35190], but perhaps

[jira] [Updated] (SPARK-35274) old hive table's all columns are read when column pruning applies in spark3.0

2021-04-29 Thread xiaoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaoli updated SPARK-35274: --- Description: I asked this question [before|hhttps://issues.apache.org/jira/browse/SPARK-35190], but perhaps

[jira] [Updated] (SPARK-35274) old hive table's all columns are read when column pruning applies in spark3.0

2021-04-29 Thread xiaoli (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaoli updated SPARK-35274: --- Description: I asked this question before, but perhaps I did not addressed question clearly, so I did not g

[jira] [Created] (SPARK-35274) old hive table's all columns are read when column pruning applies in spark3.0

2021-04-29 Thread xiaoli (Jira)
xiaoli created SPARK-35274: -- Summary: old hive table's all columns are read when column pruning applies in spark3.0 Key: SPARK-35274 URL: https://issues.apache.org/jira/browse/SPARK-35274 Project: Spark

[jira] [Updated] (SPARK-35271) After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs

2021-04-29 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-35271: --- Component/s: PySpark > After CrossValidator/TrainValidationSplit fit raised error, some backgroud >

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet") spark.sq

[jira] [Updated] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35273: Description: For example: {code:scala} spark.sql("create table t1(id int) using parquet") spark.sq

[jira] [Created] (SPARK-35273) CombineFilters support non-deterministic expressions

2021-04-29 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-35273: --- Summary: CombineFilters support non-deterministic expressions Key: SPARK-35273 URL: https://issues.apache.org/jira/browse/SPARK-35273 Project: Spark Issue Type

[jira] [Commented] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335449#comment-17335449 ] Apache Spark commented on SPARK-35084: -- User 'ocworld' has created a pull request f

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35084: Assignee: (was: Apache Spark) > [k8s] On Spark 3, jars listed in spark.jars and spark

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35084: Assignee: Apache Spark > [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packa

[jira] [Commented] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335446#comment-17335446 ] Apache Spark commented on SPARK-35084: -- User 'ocworld' has created a pull request f

[jira] [Created] (SPARK-35272) org.apache.spark.SparkException: Task not serializable

2021-04-29 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-35272: - Summary: org.apache.spark.SparkException: Task not serializable Key: SPARK-35272 URL: https://issues.apache.org/jira/browse/SPARK-35272 Project: Spark Issu

[jira] [Updated] (SPARK-35272) org.apache.spark.SparkException: Task not serializable

2021-04-29 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-35272: -- Attachment: ExceptionStack.txt > org.apache.spark.SparkException: Task not serializable >

[jira] [Commented] (SPARK-34581) BoundAttribute issue after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335426#comment-17335426 ] Apache Spark commented on SPARK-34581: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-34581) BoundAttribute issue after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335425#comment-17335425 ] Apache Spark commented on SPARK-34581: -- User 'peter-toth' has created a pull reques

[jira] [Commented] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

2021-04-29 Thread Saurabh Chawla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335374#comment-17335374 ] Saurabh Chawla commented on SPARK-26224: [~srowen] / [~hyukjin.kwon] - Any plan

[jira] [Comment Edited] (SPARK-35262) Memory leak when dataset is being persisted

2021-04-29 Thread Fu Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335343#comment-17335343 ] Fu Chen edited comment on SPARK-35262 at 4/29/21, 10:43 AM:

[jira] [Commented] (SPARK-35262) Memory leak when dataset is being persisted

2021-04-29 Thread Fu Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335343#comment-17335343 ] Fu Chen commented on SPARK-35262: - [~iamelin] This is should be a duplicate bug with SPA

[jira] [Assigned] (SPARK-35271) After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs

2021-04-29 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-35271: -- Assignee: Weichen Xu > After CrossValidator/TrainValidationSplit fit raised error, some backg

[jira] [Created] (SPARK-35271) After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs

2021-04-29 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-35271: -- Summary: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or launch new spark jobs Key: SPARK-35271 URL: https://issues.apache

[jira] [Assigned] (SPARK-35270) Remove the use of guava to fix Hadoop 3.2.2 guava conflict

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35270: Assignee: (was: Apache Spark) > Remove the use of guava to fix Hadoop 3.2.2 guava con

[jira] [Commented] (SPARK-35270) Remove the use of guava to fix Hadoop 3.2.2 guava conflict

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335256#comment-17335256 ] Apache Spark commented on SPARK-35270: -- User 'wForget' has created a pull request f

[jira] [Assigned] (SPARK-35270) Remove the use of guava to fix Hadoop 3.2.2 guava conflict

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35270: Assignee: Apache Spark > Remove the use of guava to fix Hadoop 3.2.2 guava conflict > ---

[jira] [Commented] (SPARK-35270) Remove the use of guava to fix Hadoop 3.2.2 guava conflict

2021-04-29 Thread Zhen Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335233#comment-17335233 ] Zhen Wang commented on SPARK-35270: --- Modify with reference to https://issues.apache.or

[jira] [Assigned] (SPARK-35266) Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35266: Assignee: Apache Spark > Fix an error in BenchmarkBase.scala that occurs when creating a

[jira] [Commented] (SPARK-35266) Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335192#comment-17335192 ] Apache Spark commented on SPARK-35266: -- User 'obs0811' has created a pull request f

[jira] [Assigned] (SPARK-35266) Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35266: Assignee: (was: Apache Spark) > Fix an error in BenchmarkBase.scala that occurs when

[jira] [Commented] (SPARK-35266) Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335191#comment-17335191 ] Apache Spark commented on SPARK-35266: -- User 'obs0811' has created a pull request f

[jira] [Created] (SPARK-35270) Remove the use of guava to fix Hadoop 3.2.2 guava conflict

2021-04-29 Thread Zhen Wang (Jira)
Zhen Wang created SPARK-35270: - Summary: Remove the use of guava to fix Hadoop 3.2.2 guava conflict Key: SPARK-35270 URL: https://issues.apache.org/jira/browse/SPARK-35270 Project: Spark Issue Ty

[jira] [Assigned] (SPARK-35269) Update commons-lang3 to 3.12

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35269: Assignee: Apache Spark > Update commons-lang3 to 3.12 > > >

[jira] [Assigned] (SPARK-35269) Update commons-lang3 to 3.12

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35269: Assignee: (was: Apache Spark) > Update commons-lang3 to 3.12 > --

[jira] [Commented] (SPARK-35269) Update commons-lang3 to 3.12

2021-04-29 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335186#comment-17335186 ] Apache Spark commented on SPARK-35269: -- User 'LuciferYang' has created a pull reque