[jira] [Updated] (SPARK-39757) Upgrade sbt from 1.7.0 to 1.7.1

2022-07-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-39757: Description: release notes: [https://github.com/sbt/sbt/releases] sbt 1.7.1 Bug fix * Fixes Java

[jira] [Created] (SPARK-39758) NPE on invalid patterns from the regexp functions

2022-07-12 Thread Max Gekk (Jira)
Max Gekk created SPARK-39758: Summary: NPE on invalid patterns from the regexp functions Key: SPARK-39758 URL: https://issues.apache.org/jira/browse/SPARK-39758 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-39704) Implement createIndex & dropIndex & IndexExists in JDBC (H2 dialect)

2022-07-12 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-39704. Fix Version/s: 3.4.0 Assignee: BingKun Pan Resolution: Fixed > Implement createInd

[jira] [Resolved] (SPARK-39651) Prune filter condition if compare with rand is deterministic

2022-07-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39651. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37040 [https://gith

[jira] [Assigned] (SPARK-39651) Prune filter condition if compare with rand is deterministic

2022-07-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39651: --- Assignee: jiaan.geng > Prune filter condition if compare with rand is deterministic > -

[jira] [Commented] (SPARK-39757) Upgrade sbt from 1.7.0 to 1.7.1

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566183#comment-17566183 ] Apache Spark commented on SPARK-39757: -- User 'panbingkun' has created a pull reques

[jira] [Assigned] (SPARK-39757) Upgrade sbt from 1.7.0 to 1.7.1

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39757: Assignee: (was: Apache Spark) > Upgrade sbt from 1.7.0 to 1.7.1 > ---

[jira] [Assigned] (SPARK-39757) Upgrade sbt from 1.7.0 to 1.7.1

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39757: Assignee: Apache Spark > Upgrade sbt from 1.7.0 to 1.7.1 > --

[jira] [Created] (SPARK-39757) Upgrade sbt from 1.7.0 to 1.7.1

2022-07-12 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-39757: --- Summary: Upgrade sbt from 1.7.0 to 1.7.1 Key: SPARK-39757 URL: https://issues.apache.org/jira/browse/SPARK-39757 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-39699) Make CollapseProject smarter about collection creation expressions

2022-07-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39699. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37165 [https://gith

[jira] [Commented] (SPARK-39732) pyspark.pandas.DataFrame.drop drops dataframe if axis not specified

2022-07-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566175#comment-17566175 ] Ruifeng Zheng commented on SPARK-39732: --- OK, will take a look [~hyukjin.kwon] > p

[jira] [Commented] (SPARK-39729) Why generate WholeStagecodegen for single operator?

2022-07-12 Thread xiangxiang Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566167#comment-17566167 ] xiangxiang Shen commented on SPARK-39729: - Hi [~hyukjin.kwon] , I know this con

[jira] [Commented] (SPARK-39755) SPARK_LOCAL_DIRS locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566164#comment-17566164 ] pralabhkumar commented on SPARK-39755: -- Problem seen on yarn side and the fix was

[jira] [Updated] (SPARK-39755) SPARK_LOCAL_DIRS locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-39755: - Description: In org.apache.spark.util  getConfiguredLocalDirs     {code:java} if (isRunningInYa

[jira] [Updated] (SPARK-39755) SPARK_LOCAL_DIRS locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-39755: - Description: In org.apache.spark.util  getConfiguredLocalDirs     {code:java} if (isRunningInYa

[jira] [Commented] (SPARK-39755) SPARK_LOCAL_DIRS locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566142#comment-17566142 ] pralabhkumar commented on SPARK-39755: -- [~dongjoon] Gentle ping .  > SPARK_LOCAL_D

[jira] [Updated] (SPARK-39755) SPARK_LOCAL_DIRS locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-39755: - Summary: SPARK_LOCAL_DIRS locations are not randomized in K8s (was: Spark-shuffle locations are

[jira] [Updated] (SPARK-39755) Spark-shuffle locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-39755: - Summary: Spark-shuffle locations are not randomized in K8s (was: Spark-shuffle locations are no

[jira] [Commented] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566130#comment-17566130 ] Yuming Wang commented on SPARK-39753: - [~devict] So the build side needs to be execu

[jira] [Commented] (SPARK-38901) DS V2 supports push down misc functions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566120#comment-17566120 ] Apache Spark commented on SPARK-38901: -- User 'chenzhx' has created a pull request f

[jira] [Assigned] (SPARK-38901) DS V2 supports push down misc functions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38901: Assignee: (was: Apache Spark) > DS V2 supports push down misc functions > ---

[jira] [Assigned] (SPARK-38901) DS V2 supports push down misc functions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38901: Assignee: Apache Spark > DS V2 supports push down misc functions > --

[jira] [Commented] (SPARK-38901) DS V2 supports push down misc functions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566119#comment-17566119 ] Apache Spark commented on SPARK-38901: -- User 'chenzhx' has created a pull request f

[jira] [Commented] (SPARK-39756) Better error messages for missing pandas scalars

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566115#comment-17566115 ] Apache Spark commented on SPARK-39756: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-39756) Better error messages for missing pandas scalars

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39756: Assignee: (was: Apache Spark) > Better error messages for missing pandas scalars > --

[jira] [Assigned] (SPARK-39756) Better error messages for missing pandas scalars

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39756: Assignee: Apache Spark > Better error messages for missing pandas scalars > -

[jira] [Commented] (SPARK-39756) Better error messages for missing pandas scalars

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566113#comment-17566113 ] Apache Spark commented on SPARK-39756: -- User 'xinrong-databricks' has created a pul

[jira] [Created] (SPARK-39756) Better error messages for missing pandas scalars

2022-07-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-39756: Summary: Better error messages for missing pandas scalars Key: SPARK-39756 URL: https://issues.apache.org/jira/browse/SPARK-39756 Project: Spark Issue Type:

[jira] [Commented] (SPARK-39754) Remove unused import or unnecessary {}

2022-07-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566085#comment-17566085 ] BingKun Pan commented on SPARK-39754: - [~dongjoon] Ok. > Remove unused import or un

[jira] [Resolved] (SPARK-39714) Resolve pyspark mypy part tests.

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39714. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37117 [https://gi

[jira] [Assigned] (SPARK-39714) Resolve pyspark mypy part tests.

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-39714: Assignee: bo zhao > Resolve pyspark mypy part tests. > >

[jira] [Commented] (SPARK-39747) pandas and pandas on Spark API parameter naming difference

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566079#comment-17566079 ] Hyukjin Kwon commented on SPARK-39747: -- Yeah, we should ideally fix them all > pan

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566077#comment-17566077 ] Hyukjin Kwon commented on SPARK-39743: -- No need to assign somebody. you can start w

[jira] [Commented] (SPARK-39732) pyspark.pandas.DataFrame.drop drops dataframe if axis not specified

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566076#comment-17566076 ] Hyukjin Kwon commented on SPARK-39732: -- [~podongfeng] would you mind taking a look

[jira] [Updated] (SPARK-39732) pyspark.pandas.DataFrame.drop drops dataframe if axis not specified

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-39732: - Flags: (was: Important) > pyspark.pandas.DataFrame.drop drops dataframe if axis not specified

[jira] [Resolved] (SPARK-39729) Why generate WholeStagecodegen for single operator?

2022-07-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-39729. -- Resolution: Incomplete > Why generate WholeStagecodegen for single operator? > ---

[jira] [Commented] (SPARK-39748) Include the origin logical plan for LogicalRDD if it comes from DataFrame

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566072#comment-17566072 ] Apache Spark commented on SPARK-39748: -- User 'HeartSaVioR' has created a pull reque

[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38910: Assignee: (was: Apache Spark) > Clean sparkStaging dir should before unregister() > -

[jira] [Commented] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-12 Thread shezm (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565923#comment-17565923 ] shezm commented on SPARK-39743: --- I'm Interested in this, could you assign it to me? > Una

[jira] [Reopened] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-38910: --- Assignee: (was: angerszhu) > Clean sparkStaging dir should before unregister() > -

[jira] [Commented] (SPARK-19609) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565392#comment-17565392 ] Victor Delépine commented on SPARK-19609: - Hey folks. Given that this issue was

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38910: -- Fix Version/s: (was: 3.4.0) > Clean sparkStaging dir should before unregister() >

[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir should before unregister()

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38910: Assignee: Apache Spark > Clean sparkStaging dir should before unregister() >

[jira] [Comment Edited] (SPARK-39743) Unable to set zstd compression level while writing parquet files

2022-07-12 Thread shezm (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565923#comment-17565923 ] shezm edited comment on SPARK-39743 at 7/12/22 4:28 PM: [~yeacha

[jira] [Commented] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565453#comment-17565453 ] Victor Delépine commented on SPARK-39753: - We currently work around this by doin

[jira] [Comment Edited] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565453#comment-17565453 ] Victor Delépine edited comment on SPARK-39753 at 7/12/22 12:50 PM: ---

[jira] [Resolved] (SPARK-39706) Set missing column with defaultValue as constant in `ParquetColumnVector`

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39706. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37115 [https://

[jira] [Assigned] (SPARK-39706) Set missing column with defaultValue as constant in `ParquetColumnVector`

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39706: - Assignee: Yang Jie > Set missing column with defaultValue as constant in `ParquetColumn

[jira] [Updated] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Delépine updated SPARK-39753: Description: SPARK-19609 was bulk-closed a while ago, but not fixed. I've decided to re-op

[jira] [Commented] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565387#comment-17565387 ] Victor Delépine commented on SPARK-39753: - cc [~ndimiduk] since you created the

[jira] [Created] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
Victor Delépine created SPARK-39753: --- Summary: Broadcast joins should pushdown join constraints as Filter to the larger relation Key: SPARK-39753 URL: https://issues.apache.org/jira/browse/SPARK-39753

[jira] [Commented] (SPARK-39754) Remove unused import or unnecessary {}

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565917#comment-17565917 ] Dongjoon Hyun commented on SPARK-39754: --- Hi, [~panbingkun]. For new feature or imp

[jira] [Assigned] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39754: - Assignee: BingKun Pan > Fix import issues in Scala/Java > -

[jira] [Resolved] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39754. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37166 [https://

[jira] [Updated] (SPARK-39754) Remove unused import or unnecessary {}

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39754: -- Summary: Remove unused import or unnecessary {} (was: Fix import issues in Scala/Java) > Rem

[jira] [Updated] (SPARK-39754) Remove unused import or unnecessary {}

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39754: -- Affects Version/s: 3.4.0 (was: 3.3.0) > Remove unused import or unn

[jira] [Commented] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565448#comment-17565448 ] Victor Delépine commented on SPARK-39753: - [~yumwang]  The idea would be to fil

[jira] [Assigned] (SPARK-39748) Include the origin logical plan for LogicalRDD if it comes from DataFrame

2022-07-12 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-39748: Assignee: Jungtaek Lim > Include the origin logical plan for LogicalRDD if it comes from

[jira] [Resolved] (SPARK-39748) Include the origin logical plan for LogicalRDD if it comes from DataFrame

2022-07-12 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-39748. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37161 [https://gi

[jira] [Resolved] (SPARK-39707) Add SQL reference for aggregate functions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39707. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37116 [https://

[jira] [Commented] (SPARK-39755) Spark-shuffle locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565935#comment-17565935 ] pralabhkumar commented on SPARK-39755: -- [~hyukjin.kwon]  Please comment on the same

[jira] [Assigned] (SPARK-39707) Add SQL reference for aggregate functions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39707: - Assignee: jiaan.geng > Add SQL reference for aggregate functions > ---

[jira] [Updated] (SPARK-39755) Spark-shuffle locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-39755: - Description: In org.apache.spark.util  getConfiguredLocalDirs     {code:java} if (isRunningInYa

[jira] [Commented] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

2022-07-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565410#comment-17565410 ] Yuming Wang commented on SPARK-39753: - [~devict] I do not think {{lhs.a == rhs.a}} c

[jira] [Assigned] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39754: Assignee: (was: Apache Spark) > Fix import issues in Scala/Java > ---

[jira] [Assigned] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39754: Assignee: Apache Spark > Fix import issues in Scala/Java > --

[jira] [Commented] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565425#comment-17565425 ] Apache Spark commented on SPARK-39754: -- User 'panbingkun' has created a pull reques

[jira] [Created] (SPARK-39755) Spark-shuffle locations are not randomized in K8s

2022-07-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-39755: Summary: Spark-shuffle locations are not randomized in K8s Key: SPARK-39755 URL: https://issues.apache.org/jira/browse/SPARK-39755 Project: Spark Issue Type

[jira] [Updated] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-39754: Description: Mainly focus on two issues * unnecessary braces in single import * unused import

[jira] [Resolved] (SPARK-39694) Update `${sbtProject}/test:runMain` to `${sbtProject}/Test/runMain`

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39694. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37102 [https://

[jira] [Resolved] (SPARK-39744) Add the REGEXP_INSTR function

2022-07-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-39744. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37154 [https://github.com

[jira] [Created] (SPARK-39754) Fix import issues in Scala/Java

2022-07-12 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-39754: --- Summary: Fix import issues in Scala/Java Key: SPARK-39754 URL: https://issues.apache.org/jira/browse/SPARK-39754 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-39694) Update `${sbtProject}/test:runMain` to `${sbtProject}/Test/runMain`

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39694: - Assignee: Yang Jie > Update `${sbtProject}/test:runMain` to `${sbtProject}/Test/runMain

[jira] [Resolved] (SPARK-39557) Support ARRAY, STRUCT, MAP types as DEFAULT values

2022-07-12 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-39557. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36960 [https:

[jira] [Assigned] (SPARK-39557) Support ARRAY, STRUCT, MAP types as DEFAULT values

2022-07-12 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-39557: -- Assignee: Daniel > Support ARRAY, STRUCT, MAP types as DEFAULT values > -

[jira] [Commented] (SPARK-39665) (GitHub CI) Bump workflow versions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565926#comment-17565926 ] Dongjoon Hyun commented on SPARK-39665: --- This issue is ambiguous because didn't me

[jira] [Comment Edited] (SPARK-39665) (GitHub CI) Bump workflow versions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565924#comment-17565924 ] Dongjoon Hyun edited comment on SPARK-39665 at 7/12/22 4:35 PM: --

[jira] [Resolved] (SPARK-39665) (GitHub CI) Bump workflow versions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39665. --- Resolution: Incomplete > (GitHub CI) Bump workflow versions > --

[jira] [Updated] (SPARK-39665) (GitHub CI) Bump workflow versions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39665: -- Fix Version/s: (was: 3.3.0) > (GitHub CI) Bump workflow versions > ---

[jira] [Updated] (SPARK-39665) (GitHub CI) Bump workflow versions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39665: -- Target Version/s: (was: 3.3.0) > (GitHub CI) Bump workflow versions > --

[jira] [Commented] (SPARK-39665) (GitHub CI) Bump workflow versions

2022-07-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565924#comment-17565924 ] Dongjoon Hyun commented on SPARK-39665: --- FYI, Apache Spark community has a guide f

[jira] [Assigned] (SPARK-35208) Add docs for LATERAL subqueries

2022-07-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35208: --- Assignee: Huaxin Gao > Add docs for LATERAL subqueries > --- >

[jira] [Resolved] (SPARK-35208) Add docs for LATERAL subqueries

2022-07-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35208. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37080 [https://gith

[jira] [Updated] (SPARK-39752) Spark job failed with 10M rows data with Broken pipe error

2022-07-12 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-39752: --- Fix Version/s: 3.0.2 > Spark job failed with 10M rows data with Broken pipe error >

[jira] [Updated] (SPARK-39752) Spark job failed with 10M rows data with Broken pipe error

2022-07-12 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-39752: Fix Version/s: (was: 3.0.2) > Spark job failed with 10M rows data with Broken pipe error > ---

[jira] [Updated] (SPARK-39752) Spark job failed with 10M rows data with Broken pipe error

2022-07-12 Thread SHOBHIT SHUKLA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SHOBHIT SHUKLA updated SPARK-39752: --- Attachment: Failed_spark_job_3.0.3.txt spark_job_success_3.0.2.txt > Spark j

[jira] [Created] (SPARK-39752) Spark job failed with 10M rows data with Broken pipe error

2022-07-12 Thread SHOBHIT SHUKLA (Jira)
SHOBHIT SHUKLA created SPARK-39752: -- Summary: Spark job failed with 10M rows data with Broken pipe error Key: SPARK-39752 URL: https://issues.apache.org/jira/browse/SPARK-39752 Project: Spark

[jira] [Commented] (SPARK-39699) Make CollapseProject smarter about collection creation expressions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565338#comment-17565338 ] Apache Spark commented on SPARK-39699: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-39699) Make CollapseProject smarter about collection creation expressions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39699: Assignee: Apache Spark (was: Wenchen Fan) > Make CollapseProject smarter about collectio

[jira] [Commented] (SPARK-39699) Make CollapseProject smarter about collection creation expressions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565337#comment-17565337 ] Apache Spark commented on SPARK-39699: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-39699) Make CollapseProject smarter about collection creation expressions

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39699: Assignee: Wenchen Fan (was: Apache Spark) > Make CollapseProject smarter about collectio

[jira] [Commented] (SPARK-39751) Better naming for hash aggregate key probing metric

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565321#comment-17565321 ] Apache Spark commented on SPARK-39751: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-39751) Better naming for hash aggregate key probing metric

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39751: Assignee: (was: Apache Spark) > Better naming for hash aggregate key probing metric >

[jira] [Commented] (SPARK-39751) Better naming for hash aggregate key probing metric

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565322#comment-17565322 ] Apache Spark commented on SPARK-39751: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-39751) Better naming for hash aggregate key probing metric

2022-07-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39751: Assignee: Apache Spark > Better naming for hash aggregate key probing metric > --

[jira] [Created] (SPARK-39751) Better naming for hash aggregate key probing metric

2022-07-12 Thread Cheng Su (Jira)
Cheng Su created SPARK-39751: Summary: Better naming for hash aggregate key probing metric Key: SPARK-39751 URL: https://issues.apache.org/jira/browse/SPARK-39751 Project: Spark Issue Type: Impro