[jira] [Resolved] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40085. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37524

[jira] [Assigned] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40085: Assignee: Wenchen Fan > use INTERNAL_ERROR error class instead of IllegalStateException to

[jira] [Assigned] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40096: Assignee: (was: Apache Spark) > Finalize shuffle merge slow due to connection

[jira] [Assigned] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40096: Assignee: Apache Spark > Finalize shuffle merge slow due to connection creation fails >

[jira] [Commented] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580059#comment-17580059 ] Apache Spark commented on SPARK-40096: -- User 'wankunde' has created a pull request for this issue:

[jira] [Created] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-08-15 Thread Wan Kun (Jira)
Wan Kun created SPARK-40096: --- Summary: Finalize shuffle merge slow due to connection creation fails Key: SPARK-40096 URL: https://issues.apache.org/jira/browse/SPARK-40096 Project: Spark Issue

[jira] [Commented] (SPARK-39989) Support estimate column statistics if it is foldable expression

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580046#comment-17580046 ] Apache Spark commented on SPARK-39989: -- User 'linhongliu-db' has created a pull request for this

[jira] [Commented] (SPARK-39989) Support estimate column statistics if it is foldable expression

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580045#comment-17580045 ] Apache Spark commented on SPARK-39989: -- User 'linhongliu-db' has created a pull request for this

[jira] [Commented] (SPARK-38334) Implement support for DEFAULT values for columns in tables

2022-08-15 Thread Daniel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580028#comment-17580028 ] Daniel commented on SPARK-38334: I think it is OK to declare that this feature is implemented in Apache

[jira] [Comment Edited] (SPARK-38334) Implement support for DEFAULT values for columns in tables

2022-08-15 Thread Daniel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580028#comment-17580028 ] Daniel edited comment on SPARK-38334 at 8/16/22 3:48 AM: - I think it is OK to

[jira] [Resolved] (SPARK-38334) Implement support for DEFAULT values for columns in tables

2022-08-15 Thread Daniel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel resolved SPARK-38334. Fix Version/s: 3.4.0 Resolution: Fixed > Implement support for DEFAULT values for columns in

[jira] [Commented] (SPARK-40095) sc.uiWebUrl should not throw exception when webui is disabled

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580022#comment-17580022 ] Apache Spark commented on SPARK-40095: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-40095) sc.uiWebUrl should not throw exception when webui is disabled

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40095: Assignee: (was: Apache Spark) > sc.uiWebUrl should not throw exception when webui is

[jira] [Assigned] (SPARK-40095) sc.uiWebUrl should not throw exception when webui is disabled

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40095: Assignee: Apache Spark > sc.uiWebUrl should not throw exception when webui is disabled >

[jira] [Commented] (SPARK-40095) sc.uiWebUrl should not throw exception when webui is disabled

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580021#comment-17580021 ] Apache Spark commented on SPARK-40095: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-40095) sc.uiWebUrl should not throw exception when webui is disabled

2022-08-15 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40095: - Summary: sc.uiWebUrl should not throw exception when webui is disabled Key: SPARK-40095 URL: https://issues.apache.org/jira/browse/SPARK-40095 Project: Spark

[jira] [Resolved] (SPARK-40000) Add config to toggle whether to automatically add default values for INSERTs without user-specified fields

2022-08-15 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-4. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37430

[jira] [Assigned] (SPARK-40000) Add config to toggle whether to automatically add default values for INSERTs without user-specified fields

2022-08-15 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-4: -- Assignee: Daniel > Add config to toggle whether to automatically add default values

[jira] [Commented] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580003#comment-17580003 ] Apache Spark commented on SPARK-40094: -- User 'wangshengjie123' has created a pull request for this

[jira] [Assigned] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40094: Assignee: (was: Apache Spark) > Send TaskEnd event when task failed with

[jira] [Assigned] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40094: Assignee: Apache Spark > Send TaskEnd event when task failed with

[jira] [Commented] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580002#comment-17580002 ] Apache Spark commented on SPARK-40094: -- User 'wangshengjie123' has created a pull request for this

[jira] [Commented] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-15 Thread wangshengjie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1757#comment-1757 ] wangshengjie commented on SPARK-40094: -- I'm working on this, a pr will be submitted later, thanks.  

[jira] [Created] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

2022-08-15 Thread wangshengjie (Jira)
wangshengjie created SPARK-40094: Summary: Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation Key: SPARK-40094

[jira] [Created] (SPARK-40093) is kubernetes jar required if not using that executor?

2022-08-15 Thread t oo (Jira)
t oo created SPARK-40093: Summary: is kubernetes jar required if not using that executor? Key: SPARK-40093 URL: https://issues.apache.org/jira/browse/SPARK-40093 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-40093) is kubernetes jar required if not using that executor?

2022-08-15 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-40093: - Description: my docker file is very big with pyspark can i remove dis files below if i don't use 'kubernetes

[jira] [Created] (SPARK-40092) is breeze required if not using ML?

2022-08-15 Thread t oo (Jira)
t oo created SPARK-40092: Summary: is breeze required if not using ML? Key: SPARK-40092 URL: https://issues.apache.org/jira/browse/SPARK-40092 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-40092) is breeze required if not using ML?

2022-08-15 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-40092: - Description: my docker file is very big with pyspark can i remove dis files below if i don't use 'ML'? 14M    

[jira] [Created] (SPARK-40091) is rocksdbjni required if not using streaming?

2022-08-15 Thread t oo (Jira)
t oo created SPARK-40091: Summary: is rocksdbjni required if not using streaming? Key: SPARK-40091 URL: https://issues.apache.org/jira/browse/SPARK-40091 Project: Spark Issue Type: Question

[jira] [Commented] (SPARK-40089) Doring of at least Decimal(20, 2) fails for some values near the max.

2022-08-15 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579984#comment-17579984 ] XiDuo You commented on SPARK-40089: --- thank you [~revans2] for reporting the issue, I can reproduce it

[jira] [Resolved] (SPARK-40009) Add missing doc string info to DataFrame API

2022-08-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40009. -- Fix Version/s: 3.4.0 Assignee: Khalid Mammadov Resolution: Fixed Resolved by

[jira] [Assigned] (SPARK-40077) Make pyspark.context examples self-contained

2022-08-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40077: Assignee: Ruifeng Zheng > Make pyspark.context examples self-contained >

[jira] [Resolved] (SPARK-40077) Make pyspark.context examples self-contained

2022-08-15 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40077. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37517

[jira] [Commented] (SPARK-37442) In AQE, wrong InMemoryRelation size estimation causes "Cannot broadcast the table that is larger than 8GB: 8 GB" failure

2022-08-15 Thread EmmaYang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579969#comment-17579969 ] EmmaYang commented on SPARK-37442: -- Hello, I exactly have this issue. and I am usign spark2.4 so my

[jira] [Resolved] (SPARK-40066) ANSI mode: always return null on invalid access to map column

2022-08-15 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-40066. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37503

[jira] [Resolved] (SPARK-40090) Upgrade to Py4J 0.10.9.7

2022-08-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-40090. -- Resolution: Duplicate > Upgrade to Py4J 0.10.9.7 > > >

[jira] [Assigned] (SPARK-40090) Upgrade to Py4J 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40090: Assignee: (was: Apache Spark) > Upgrade to Py4J 0.10.9.7 >

[jira] [Commented] (SPARK-40090) Upgrade to Py4J 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579930#comment-17579930 ] Apache Spark commented on SPARK-40090: -- User 'xinrong-meng' has created a pull request for this

[jira] [Assigned] (SPARK-40090) Upgrade to Py4J 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40090: Assignee: Apache Spark > Upgrade to Py4J 0.10.9.7 > > >

[jira] [Commented] (SPARK-40090) Upgrade to Py4J 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579929#comment-17579929 ] Apache Spark commented on SPARK-40090: -- User 'xinrong-meng' has created a pull request for this

[jira] [Created] (SPARK-40090) Upgrade to Py4J 0.10.9.7

2022-08-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-40090: Summary: Upgrade to Py4J 0.10.9.7 Key: SPARK-40090 URL: https://issues.apache.org/jira/browse/SPARK-40090 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-40089) Doring of at least Decimal(20, 2) fails for some values near the max.

2022-08-15 Thread Robert Joseph Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579897#comment-17579897 ] Robert Joseph Evans commented on SPARK-40089: - Looking at the code it appears that the

[jira] [Commented] (SPARK-40089) Doring of at least Decimal(20, 2) fails for some values near the max.

2022-08-15 Thread Robert Joseph Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579892#comment-17579892 ] Robert Joseph Evans commented on SPARK-40089: - It sure looks like it is related to the

[jira] [Commented] (SPARK-40089) Doring of at least Decimal(20, 2) fails for some values near the max.

2022-08-15 Thread Robert Joseph Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579887#comment-17579887 ] Robert Joseph Evans commented on SPARK-40089: - I have been trying to debug this and it does

[jira] [Updated] (SPARK-40089) Doring of at least Decimal(20, 2) fails for some values near the max.

2022-08-15 Thread Robert Joseph Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated SPARK-40089: Attachment: input.parquet > Doring of at least Decimal(20, 2) fails for some

[jira] [Created] (SPARK-40089) Doring of at least Decimal(20, 2) fails for some values near the max.

2022-08-15 Thread Robert Joseph Evans (Jira)
Robert Joseph Evans created SPARK-40089: --- Summary: Doring of at least Decimal(20, 2) fails for some values near the max. Key: SPARK-40089 URL: https://issues.apache.org/jira/browse/SPARK-40089

[jira] [Commented] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

2022-08-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579845#comment-17579845 ] Dongjoon Hyun commented on SPARK-3: --- BTW, SPARK-35781 has more backgrounds. - LevelDB,

[jira] [Commented] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

2022-08-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579839#comment-17579839 ] Dongjoon Hyun commented on SPARK-3: --- Yes, [~tgraves]. LevelDB is too ancient and has

[jira] [Resolved] (SPARK-40064) Use V2 Filter in SupportsOverwrite

2022-08-15 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-40064. Fix Version/s: 3.4.0 Assignee: Huaxin Gao Resolution: Fixed > Use V2 Filter in

[jira] [Created] (SPARK-40088) Add SparkPlanWIthAQESuite

2022-08-15 Thread Kazuyuki Tanimura (Jira)
Kazuyuki Tanimura created SPARK-40088: - Summary: Add SparkPlanWIthAQESuite Key: SPARK-40088 URL: https://issues.apache.org/jira/browse/SPARK-40088 Project: Spark Issue Type: Test

[jira] [Assigned] (SPARK-40087) Support multiple Column drop in R

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40087: Assignee: Apache Spark > Support multiple Column drop in R >

[jira] [Assigned] (SPARK-40087) Support multiple Column drop in R

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40087: Assignee: (was: Apache Spark) > Support multiple Column drop in R >

[jira] [Commented] (SPARK-40087) Support multiple Column drop in R

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579826#comment-17579826 ] Apache Spark commented on SPARK-40087: -- User 'santosh-d3vpl3x' has created a pull request for this

[jira] [Created] (SPARK-40087) Support multiple Column drop in R

2022-08-15 Thread Santosh Pingale (Jira)
Santosh Pingale created SPARK-40087: --- Summary: Support multiple Column drop in R Key: SPARK-40087 URL: https://issues.apache.org/jira/browse/SPARK-40087 Project: Spark Issue Type: New

[jira] [Assigned] (SPARK-40086) Improve AliasAwareOutputPartitioning to take all aliases into account

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40086: Assignee: (was: Apache Spark) > Improve AliasAwareOutputPartitioning to take all

[jira] [Commented] (SPARK-40086) Improve AliasAwareOutputPartitioning to take all aliases into account

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579814#comment-17579814 ] Apache Spark commented on SPARK-40086: -- User 'peter-toth' has created a pull request for this

[jira] [Assigned] (SPARK-40086) Improve AliasAwareOutputPartitioning to take all aliases into account

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40086: Assignee: Apache Spark > Improve AliasAwareOutputPartitioning to take all aliases into

[jira] [Created] (SPARK-40086) Improve AliasAwareOutputPartitioning to take all aliases into account

2022-08-15 Thread Peter Toth (Jira)
Peter Toth created SPARK-40086: -- Summary: Improve AliasAwareOutputPartitioning to take all aliases into account Key: SPARK-40086 URL: https://issues.apache.org/jira/browse/SPARK-40086 Project: Spark

[jira] [Commented] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579749#comment-17579749 ] Apache Spark commented on SPARK-40085: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40085: Assignee: Apache Spark > use INTERNAL_ERROR error class instead of IllegalStateException

[jira] [Assigned] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40085: Assignee: (was: Apache Spark) > use INTERNAL_ERROR error class instead of

[jira] [Commented] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579748#comment-17579748 ] Apache Spark commented on SPARK-40085: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-40085) use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs

2022-08-15 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-40085: --- Summary: use INTERNAL_ERROR error class instead of IllegalStateException to indicate bugs Key: SPARK-40085 URL: https://issues.apache.org/jira/browse/SPARK-40085

[jira] [Commented] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

2022-08-15 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579738#comment-17579738 ] Thomas Graves commented on SPARK-3: --- Just curious does rocksdb give us some particular benefit

[jira] [Assigned] (SPARK-40058) Avoid filter twice in HadoopFSUtils

2022-08-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-40058: Assignee: ZiyueGuan > Avoid filter twice in HadoopFSUtils >

[jira] [Resolved] (SPARK-40058) Avoid filter twice in HadoopFSUtils

2022-08-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40058. -- Fix Version/s: 3.4.0 Resolution: Fixed Resolved by

[jira] [Resolved] (SPARK-40035) Avoid apply filter twice when listing files

2022-08-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40035. -- Resolution: Duplicate > Avoid apply filter twice when listing files >

[jira] [Assigned] (SPARK-39887) Expression transform error

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39887: --- Assignee: Peter Toth > Expression transform error > -- > >

[jira] [Updated] (SPARK-39887) Expression transform error

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-39887: Fix Version/s: 3.4.0 3.3.1 3.2.3 > Expression transform

[jira] [Resolved] (SPARK-39887) Expression transform error

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39887. - Fix Version/s: 3.1.4 Resolution: Fixed Issue resolved by pull request 37496

[jira] [Resolved] (SPARK-39982) StructType.fromJson method missing documentation

2022-08-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-39982. -- Fix Version/s: 3.4.0 Resolution: Fixed Resolved by

[jira] [Assigned] (SPARK-39982) StructType.fromJson method missing documentation

2022-08-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-39982: Assignee: Khalid Mammadov > StructType.fromJson method missing documentation >

[jira] [Resolved] (SPARK-40019) Refactor comment of ArrayType

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40019. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37453

[jira] [Assigned] (SPARK-40019) Refactor comment of ArrayType

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40019: --- Assignee: angerszhu > Refactor comment of ArrayType > - > >

[jira] [Resolved] (SPARK-40073) Should Use `connector/${moduleName}` instead of `external/${moduleName}`

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40073. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37512

[jira] [Comment Edited] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread wangshengjie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579692#comment-17579692 ] wangshengjie edited comment on SPARK-40083 at 8/15/22 1:03 PM: --- Maybe i

[jira] [Assigned] (SPARK-40073) Should Use `connector/${moduleName}` instead of `external/${moduleName}`

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40073: --- Assignee: Yang Jie > Should Use `connector/${moduleName}` instead of

[jira] [Commented] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread wangshengjie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579692#comment-17579692 ] wangshengjie commented on SPARK-40083: -- Maybe i should add expire policy for push merge shuffle

[jira] [Resolved] (SPARK-39976) NULL check in ArrayIntersect adds extraneous null from first param

2022-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39976. - Fix Version/s: 3.4.0 3.3.1 Assignee: angerszhu Resolution:

[jira] [Commented] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579685#comment-17579685 ] Apache Spark commented on SPARK-40084: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40084: Assignee: (was: Apache Spark) > Upgrade Py4J from 0.10.9.5 to 0.10.9.7 >

[jira] [Assigned] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40084: Assignee: Apache Spark > Upgrade Py4J from 0.10.9.5 to 0.10.9.7 >

[jira] [Created] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-08-15 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-40084: --- Summary: Upgrade Py4J from 0.10.9.5 to 0.10.9.7 Key: SPARK-40084 URL: https://issues.apache.org/jira/browse/SPARK-40084 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Penglei Shi updated SPARK-40082: Description: In condition of push-based shuffle being enabled and speculative tasks existing, a

[jira] [Assigned] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40083: Assignee: (was: Apache Spark) > Add shuffle index cache expire time policy to avoid

[jira] [Assigned] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40083: Assignee: Apache Spark > Add shuffle index cache expire time policy to avoid unused

[jira] [Commented] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579659#comment-17579659 ] Apache Spark commented on SPARK-40083: -- User 'wangshengjie123' has created a pull request for this

[jira] [Commented] (SPARK-40078) Make pyspark.sql.column examples self-contained

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579658#comment-17579658 ] Apache Spark commented on SPARK-40078: -- User 'dcoliversun' has created a pull request for this

[jira] [Assigned] (SPARK-40078) Make pyspark.sql.column examples self-contained

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40078: Assignee: Apache Spark > Make pyspark.sql.column examples self-contained >

[jira] [Assigned] (SPARK-40078) Make pyspark.sql.column examples self-contained

2022-08-15 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40078: Assignee: (was: Apache Spark) > Make pyspark.sql.column examples self-contained >

[jira] [Commented] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread wangshengjie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579651#comment-17579651 ] wangshengjie commented on SPARK-40083: -- I'm working on this, a pr will be submitted later. > Add

[jira] [Created] (SPARK-40083) Add shuffle index cache expire time policy to avoid unused continuous memory consumption

2022-08-15 Thread wangshengjie (Jira)
wangshengjie created SPARK-40083: Summary: Add shuffle index cache expire time policy to avoid unused continuous memory consumption Key: SPARK-40083 URL: https://issues.apache.org/jira/browse/SPARK-40083

[jira] [Resolved] (SPARK-40079) Add Imputer inputCols validation for empty input case

2022-08-15 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-40079. Fix Version/s: 3.1.4 3.4.0 3.3.1 3.2.3

[jira] [Commented] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579612#comment-17579612 ] Penglei Shi commented on SPARK-40082: - ping [~mshen]  > DAGScheduler may not schduler new stage in

[jira] [Updated] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Penglei Shi updated SPARK-40082: Description: In condition of push-based shuffle being enabled and speculative tasks existing, a

[jira] [Updated] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Penglei Shi updated SPARK-40082: Description: In condition of push-based shuffle being enabled and speculative tasks existing, a

[jira] [Updated] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Penglei Shi updated SPARK-40082: Attachment: missParentStages.png > DAGScheduler may not schduler new stage in condition of

[jira] [Updated] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Penglei Shi updated SPARK-40082: Attachment: submitMissingTasks.png > DAGScheduler may not schduler new stage in condition of

[jira] [Updated] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Penglei Shi updated SPARK-40082: Attachment: shuffleMergeFinalized.png > DAGScheduler may not schduler new stage in condition of

[jira] [Created] (SPARK-40082) DAGScheduler may not schduler new stage in condition of push-based shuffle enabled

2022-08-15 Thread Penglei Shi (Jira)
Penglei Shi created SPARK-40082: --- Summary: DAGScheduler may not schduler new stage in condition of push-based shuffle enabled Key: SPARK-40082 URL: https://issues.apache.org/jira/browse/SPARK-40082

  1   2   >