[jira] [Resolved] (SPARK-43209) Migrate Expression errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43209. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40869

[jira] [Assigned] (SPARK-43209) Migrate Expression errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43209: - Assignee: Haejoon Lee > Migrate Expression errors into error class >

[jira] [Updated] (SPARK-43240) df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow]

[ https://issues.apache.org/jira/browse/SPARK-43240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-43240: --- Affects Version/s: 3.3.2 (was: 3.2.2) > df.describe() method may- return wrong

[jira] [Created] (SPARK-43245) Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64' type of Index.

Haejoon Lee created SPARK-43245: --- Summary: Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64' type of Index. Key: SPARK-43245 URL: https://issues.apache.org/jira/browse/SPARK-43245

[jira] [Assigned] (SPARK-43210) Introduce PySparkAssersionError

[ https://issues.apache.org/jira/browse/SPARK-43210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43210: - Assignee: Haejoon Lee > Introduce PySparkAssersionError >

[jira] [Resolved] (SPARK-43210) Introduce PySparkAssersionError

[ https://issues.apache.org/jira/browse/SPARK-43210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43210. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40868

[jira] [Updated] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

[ https://issues.apache.org/jira/browse/SPARK-43113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43113: - Fix Version/s: 3.3.3 > Codegen error when full outer join's bound condition has multiple

[jira] [Resolved] (SPARK-42945) Support PYSPARK_JVM_STACKTRACE_ENABLED in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-42945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42945. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40575

[jira] [Resolved] (SPARK-43212) Migrate Structured Streaming errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43212. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40880

[jira] [Assigned] (SPARK-43212) Migrate Structured Streaming errors into error class

[ https://issues.apache.org/jira/browse/SPARK-43212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43212: Assignee: Haejoon Lee > Migrate Structured Streaming errors into error class >

[jira] [Assigned] (SPARK-43239) Remove null_counts from info()

[ https://issues.apache.org/jira/browse/SPARK-43239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43239: Assignee: Bjørn Jørgensen > Remove null_counts from info() >

[jira] [Resolved] (SPARK-43239) Remove null_counts from info()

[ https://issues.apache.org/jira/browse/SPARK-43239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43239. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40913

[jira] [Commented] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715525#comment-17715525 ] Adam Binford commented on SPARK-43244: -- Read through [https://github.com/apache/spark/pull/30344]

[jira] [Updated] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Binford updated SPARK-43244: - Description: We noticed in one of our production stateful streaming jobs using RocksDB that an

[jira] [Updated] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Binford updated SPARK-43244: - Description: We noticed in one of our production stateful streaming jobs using RocksDB that an

[jira] [Updated] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Binford updated SPARK-43244: - Description: We noticed in one of our production stateful streaming jobs using RocksDB that an

[jira] [Commented] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

[ https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715495#comment-17715495 ] Adam Binford commented on SPARK-43244: -- [~kabhwan] curious your thoughts on this. Seems somewhat

[jira] [Created] (SPARK-43244) RocksDB State Store can accumulate unbounded native memory

Adam Binford created SPARK-43244: Summary: RocksDB State Store can accumulate unbounded native memory Key: SPARK-43244 URL: https://issues.apache.org/jira/browse/SPARK-43244 Project: Spark

[jira] [Created] (SPARK-43243) Add Level param to df.printSchema for Python API

Khalid Mammadov created SPARK-43243: --- Summary: Add Level param to df.printSchema for Python API Key: SPARK-43243 URL: https://issues.apache.org/jira/browse/SPARK-43243 Project: Spark Issue

[jira] [Updated] (SPARK-43008) Upgrade jodatime to 2.12.5

[ https://issues.apache.org/jira/browse/SPARK-43008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-43008: - Priority: Minor (was: Major) > Upgrade jodatime to 2.12.5 > -- > >

[jira] [Resolved] (SPARK-43008) Upgrade jodatime to 2.12.5

[ https://issues.apache.org/jira/browse/SPARK-43008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-43008. -- Fix Version/s: 3.5.0 Assignee: Yang Jie Resolution: Fixed Resolved by

[jira] [Updated] (SPARK-43242) diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId

[ https://issues.apache.org/jira/browse/SPARK-43242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhang Liang updated SPARK-43242: Summary: diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId

[jira] [Updated] (SPARK-43242) shuffle diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId

[ https://issues.apache.org/jira/browse/SPARK-43242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhang Liang updated SPARK-43242: Description: Some of our spark app throw "Unexpected type of BlockId" exception as shown below

[jira] [Updated] (SPARK-43242) shuffle diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId

[ https://issues.apache.org/jira/browse/SPARK-43242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhang Liang updated SPARK-43242: Description: Some of our spark app throw "Unexpected type of BlockId" exception as shown below

[jira] [Created] (SPARK-43242) shuffle diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId

Zhang Liang created SPARK-43242: --- Summary: shuffle diagnoseCorruption should not throw Unexpected type of BlockId for ShuffleBlockBatchId Key: SPARK-43242 URL: https://issues.apache.org/jira/browse/SPARK-43242

[jira] [Commented] (SPARK-43232) Improve ObjectHashAggregateExec performance for high cardinality

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715417#comment-17715417 ] ASF GitHub Bot commented on SPARK-43232: User 'ulysses-you' has created a pull request for this

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance for high cardinality

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has three preformance issues: - heavy overhead of scala

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance for high cardinality

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Summary: Improve ObjectHashAggregateExec performance for high cardinality (was: Improve

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance with high cardinality

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Summary: Improve ObjectHashAggregateExec performance with high cardinality (was: Improve

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has three preformance issues: - heavy overhead of scala

[jira] [Updated] (SPARK-43232) Improve ObjectHashAggregateExec performance

[ https://issues.apache.org/jira/browse/SPARK-43232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-43232: -- Description: The `ObjectHashAggregateExec` has three preformance issues: - heavy overhead of scala

[jira] [Created] (SPARK-43241) MultiIndex.append not checking names for equality

Haejoon Lee created SPARK-43241: --- Summary: MultiIndex.append not checking names for equality Key: SPARK-43241 URL: https://issues.apache.org/jira/browse/SPARK-43241 Project: Spark Issue Type: