[jira] [Resolved] (SPARK-48757) Make `IndexShuffleBlockResolver` have explicit constructors

2024-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48757. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47148 [https://

[jira] [Created] (SPARK-48758) Race condition between executor registration and heartbeat

2024-06-28 Thread MJ Deng (Jira)
MJ Deng created SPARK-48758: --- Summary: Race condition between executor registration and heartbeat Key: SPARK-48758 URL: https://issues.apache.org/jira/browse/SPARK-48758 Project: Spark Issue Type:

[jira] [Updated] (SPARK-48757) Make `IndexShuffleBlockResolver` have explicit constructors

2024-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48757: -- Description: Although `IndexShuffleBlockResolver` is `private` and there is no contract to ke

[jira] [Updated] (SPARK-48757) Make `IndexShuffleBlockResolver` have explicit constructors

2024-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48757: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Make `IndexShuffleBlock

[jira] [Assigned] (SPARK-48757) Make `IndexShuffleBlockResolver` have explicit constructors

2024-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48757: - Assignee: Dongjoon Hyun > Make `IndexShuffleBlockResolver` have explicit constructors >

[jira] [Created] (SPARK-48757) Make `IndexShuffleBlockResolver` have explicit constructors

2024-06-28 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48757: - Summary: Make `IndexShuffleBlockResolver` have explicit constructors Key: SPARK-48757 URL: https://issues.apache.org/jira/browse/SPARK-48757 Project: Spark

[jira] [Created] (SPARK-48756) Add support for `df.debug()`

2024-06-28 Thread Martin Grund (Jira)
Martin Grund created SPARK-48756: Summary: Add support for `df.debug()` Key: SPARK-48756 URL: https://issues.apache.org/jira/browse/SPARK-48756 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-48755) [Python State V2] Base implementation and ValueState support

2024-06-28 Thread Bo Gao (Jira)
Bo Gao created SPARK-48755: -- Summary: [Python State V2] Base implementation and ValueState support Key: SPARK-48755 URL: https://issues.apache.org/jira/browse/SPARK-48755 Project: Spark Issue Type:

[jira] [Updated] (SPARK-48754) Improve code towards productionizing the go library

2024-06-28 Thread Peter Kristensen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kristensen updated SPARK-48754: - Description: The readme states: "is highly experimental and should not be used in any p

[jira] [Updated] (SPARK-48754) Improve code towards productionizing the go library

2024-06-28 Thread Peter Kristensen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kristensen updated SPARK-48754: - Labels: pull-request-available (was: ) > Improve code towards productionizing the go li

[jira] [Created] (SPARK-48754) Improve code towards productionizing the go library

2024-06-28 Thread Peter Kristensen (Jira)
Peter Kristensen created SPARK-48754: Summary: Improve code towards productionizing the go library Key: SPARK-48754 URL: https://issues.apache.org/jira/browse/SPARK-48754 Project: Spark I

[jira] [Updated] (SPARK-48753) Upgrade arrow in the go library

2024-06-28 Thread Peter Kristensen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kristensen updated SPARK-48753: - Labels: pull-request-available (was: ) > Upgrade arrow in the go library >

[jira] [Created] (SPARK-48753) Upgrade arrow in the go library

2024-06-28 Thread Peter Kristensen (Jira)
Peter Kristensen created SPARK-48753: Summary: Upgrade arrow in the go library Key: SPARK-48753 URL: https://issues.apache.org/jira/browse/SPARK-48753 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-48752) Introduce `pyspark.logging` for improved structured logging for PySpark

2024-06-28 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-48752: --- Summary: Introduce `pyspark.logging` for improved structured logging for PySpark Key: SPARK-48752 URL: https://issues.apache.org/jira/browse/SPARK-48752 Project: Spark

[jira] [Updated] (SPARK-48751) Re-balance`pyspark-pandas-connect` tests on GA

2024-06-28 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-48751: Summary: Re-balance`pyspark-pandas-connect` tests on GA (was: Re-balance`pyspark-pandas-connect`

[jira] [Updated] (SPARK-48751) Re-balance`pyspark-pandas-connect` test on GA

2024-06-28 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-48751: Summary: Re-balance`pyspark-pandas-connect` test on GA (was: Balance `pyspark-pandas-connect` tes

[jira] [Created] (SPARK-48751) Balance `pyspark-pandas-connect` test on GA

2024-06-28 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48751: --- Summary: Balance `pyspark-pandas-connect` test on GA Key: SPARK-48751 URL: https://issues.apache.org/jira/browse/SPARK-48751 Project: Spark Issue Type: Improve

[jira] [Resolved] (SPARK-48673) Scheduling Across Applications in k8s mode

2024-06-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48673. -- Resolution: Information Provided Please use the dev or user mailing lists for questions https://spark

[jira] [Updated] (SPARK-48673) Scheduling Across Applications in k8s mode

2024-06-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-48673: - Issue Type: Question (was: Improvement) > Scheduling Across Applications in k8s mode > ---

[jira] [Updated] (SPARK-48673) Scheduling Across Applications in k8s mode

2024-06-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-48673: - Priority: Trivial (was: Blocker) > Scheduling Across Applications in k8s mode > --

[jira] [Resolved] (SPARK-48460) Spark ORC writer generates incorrect meta information(min, max)

2024-06-28 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-48460. Resolution: Not A Bug It is clean design decision in ORC. See the ORC implementat

[jira] [Commented] (SPARK-48460) Spark ORC writer generates incorrect meta information(min, max)

2024-06-28 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860772#comment-17860772 ] Attila Zsolt Piros commented on SPARK-48460: This is coming from ORC directl

[jira] [Updated] (SPARK-48750) AQEPropagateEmptyRelation convert broadcast query stage plan to empty relation causing error

2024-06-28 Thread Yi Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhu updated SPARK-48750: --- Description: {code:java}     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.

[jira] [Updated] (SPARK-48750) AQEPropagateEmptyRelation convert broadcast query stage plan to empty relation causing error

2024-06-28 Thread Yi Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Zhu updated SPARK-48750: --- Description: {code:java}     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.

[jira] [Created] (SPARK-48750) AQEPropagateEmptyRelation convert broadcast query stage plan to empty relation causing error

2024-06-28 Thread Yi Zhu (Jira)
Yi Zhu created SPARK-48750: -- Summary: AQEPropagateEmptyRelation convert broadcast query stage plan to empty relation causing error Key: SPARK-48750 URL: https://issues.apache.org/jira/browse/SPARK-48750 Proj

[jira] [Created] (SPARK-48749) Simplify UnaryPositive with RuntimeReplacable

2024-06-28 Thread Kent Yao (Jira)
Kent Yao created SPARK-48749: Summary: Simplify UnaryPositive with RuntimeReplacable Key: SPARK-48749 URL: https://issues.apache.org/jira/browse/SPARK-48749 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-48586) Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()

2024-06-28 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-48586: - Fix Version/s: 3.5.2 > Remove lock acquisition in doMaintenance() by making a deep copy of file

[jira] [Commented] (SPARK-48307) InlineCTE should keep not-inlined relations in the original WithCTE node

2024-06-28 Thread ci-cassandra.apache.org (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860739#comment-17860739 ] ci-cassandra.apache.org commented on SPARK-48307: - User 'cloud-fan' has

[jira] [Created] (SPARK-48748) Cache numChars in UTF8String

2024-06-28 Thread Jira
Uroš Bojanić created SPARK-48748: Summary: Cache numChars in UTF8String Key: SPARK-48748 URL: https://issues.apache.org/jira/browse/SPARK-48748 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-48745) Remove unnecessary installation `miniconda` from `build_and_test.yml`

2024-06-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48745: Assignee: BingKun Pan > Remove unnecessary installation `miniconda` from `build_and_test.

[jira] [Resolved] (SPARK-48745) Remove unnecessary installation `miniconda` from `build_and_test.yml`

2024-06-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48745. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47138 [https://gi

[jira] [Resolved] (SPARK-48746) Avoid using global temp view in foreachBatch test case

2024-06-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48746. -- Fix Version/s: 4.0.0 Assignee: Hyukjin Kwon Resolution: Fixed Fixed in https:/