[jira] [Commented] (SPARK-37145) Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer API

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491281#comment-17491281 ] Apache Spark commented on SPARK-37145: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-37145) Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer API

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491280#comment-17491280 ] Apache Spark commented on SPARK-37145: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-36061) Add Volcano feature step

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491279#comment-17491279 ] Apache Spark commented on SPARK-36061: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-36061) Add Volcano feature step

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491278#comment-17491278 ] Apache Spark commented on SPARK-36061: -- User 'Yikun' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-20236) Overwrite a partitioned data source table should only overwrite related partitions

2022-02-11 Thread Anand Kannachikandy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491265#comment-17491265 ] Anand Kannachikandy edited comment on SPARK-20236 at 2/12/22, 5:48 AM:

[jira] [Comment Edited] (SPARK-20236) Overwrite a partitioned data source table should only overwrite related partitions

2022-02-11 Thread Anand Kannachikandy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491265#comment-17491265 ] Anand Kannachikandy edited comment on SPARK-20236 at 2/12/22, 5:48 AM:

[jira] [Commented] (SPARK-20236) Overwrite a partitioned data source table should only overwrite related partitions

2022-02-11 Thread Anand Kannachikandy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491265#comment-17491265 ] Anand Kannachikandy commented on SPARK-20236: - [~cloud_fan]  Have anyone tested this in HDP

[jira] [Commented] (SPARK-37960) A new framework to represent catalyst expressions in DS v2 APIs

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491236#comment-17491236 ] Apache Spark commented on SPARK-37960: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-37960) A new framework to represent catalyst expressions in DS v2 APIs

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491235#comment-17491235 ] Apache Spark commented on SPARK-37960: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-11 Thread Marnix van den Broek (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491170#comment-17491170 ] Marnix van den Broek commented on SPARK-38167: -- After getting some help from the community

[jira] [Created] (SPARK-38192) Fix the risk of resource leak in RocksDBSuite on Apple Silicon

2022-02-11 Thread Yang Jie (Jira)
Yang Jie created SPARK-38192: Summary: Fix the risk of resource leak in RocksDBSuite on Apple Silicon Key: SPARK-38192 URL: https://issues.apache.org/jira/browse/SPARK-38192 Project: Spark

[jira] [Resolved] (SPARK-38185) Fix data incorrect if aggregate function is empty

2022-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38185. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-38185) Fix data incorrect if aggregate function is empty

2022-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38185: --- Assignee: XiDuo You > Fix data incorrect if aggregate function is empty >

[jira] [Commented] (SPARK-38191) The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol.

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490894#comment-17490894 ] Apache Spark commented on SPARK-38191: -- User 'weixiuli' has created a pull request for this issue:

[jira] [Commented] (SPARK-38191) The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol.

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490892#comment-17490892 ] Apache Spark commented on SPARK-38191: -- User 'weixiuli' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38191) The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol.

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38191: Assignee: Apache Spark > The staging directory of write job only needs to be initialized

[jira] [Assigned] (SPARK-38191) The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol.

2022-02-11 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38191: Assignee: (was: Apache Spark) > The staging directory of write job only needs to be

[jira] [Created] (SPARK-38191) The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol.

2022-02-11 Thread weixiuli (Jira)
weixiuli created SPARK-38191: Summary: The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol. Key: SPARK-38191 URL: https://issues.apache.org/jira/browse/SPARK-38191

[jira] [Commented] (SPARK-38174) SparkR documentation build fails in CI

2022-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490878#comment-17490878 ] Hyukjin Kwon commented on SPARK-38174: -- Awesome!!! > SparkR documentation build fails in CI >

[jira] [Commented] (SPARK-38174) SparkR documentation build fails in CI

2022-02-11 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490877#comment-17490877 ] Maciej Szymkiewicz commented on SPARK-38174: Done some more digging, and I believe it has

[jira] [Comment Edited] (SPARK-30442) Write mode ignored when using CodecStreams

2022-02-11 Thread Paul ONeill (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490869#comment-17490869 ] Paul ONeill edited comment on SPARK-30442 at 2/11/22, 11:42 AM: Hey

[jira] [Commented] (SPARK-30442) Write mode ignored when using CodecStreams

2022-02-11 Thread Paul ONeill (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490869#comment-17490869 ] Paul ONeill commented on SPARK-30442: - Hey [~hyukjin.kwon] , [~abhimadav] 's comment seems to match

[jira] [Comment Edited] (SPARK-30442) Write mode ignored when using CodecStreams

2022-02-11 Thread Paul ONeill (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490869#comment-17490869 ] Paul ONeill edited comment on SPARK-30442 at 2/11/22, 11:43 AM: Hey

[jira] [Commented] (SPARK-30442) Write mode ignored when using CodecStreams

2022-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490865#comment-17490865 ] Hyukjin Kwon commented on SPARK-30442: -- Can you use overwrite mode in DataFrameWriter options? It

[jira] [Commented] (SPARK-30442) Write mode ignored when using CodecStreams

2022-02-11 Thread Paul ONeill (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490863#comment-17490863 ] Paul ONeill commented on SPARK-30442: - [~hyukjin.kwon]  Is this not being worked on? It's still an

[jira] [Updated] (SPARK-38189) Support preemption scheduling (Introduce priorityClass) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38189: Summary: Support preemption scheduling (Introduce priorityClass) with volcano implementations

[jira] [Created] (SPARK-38190) Support fair-share scheduling

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38190: --- Summary: Support fair-share scheduling Key: SPARK-38190 URL: https://issues.apache.org/jira/browse/SPARK-38190 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-38189) Preemption scheduling (Introduce priorityClass) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38189: --- Summary: Preemption scheduling (Introduce priorityClass) with volcano implementations Key: SPARK-38189 URL: https://issues.apache.org/jira/browse/SPARK-38189 Project:

[jira] [Created] (SPARK-38188) Support queue scheduling (Introduce queue) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38188: --- Summary: Support queue scheduling (Introduce queue) with volcano implementations Key: SPARK-38188 URL: https://issues.apache.org/jira/browse/SPARK-38188 Project: Spark

[jira] [Updated] (SPARK-38187) Support resource reservation (Introduce minCPU/minMemory) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38187: Summary: Support resource reservation (Introduce minCPU/minMemory) with volcano implementations

[jira] [Updated] (SPARK-38187) Resource reservation (Introduce minCPU/minMemory) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38187: Parent: SPARK-36057 Issue Type: Sub-task (was: Bug) > Resource reservation (Introduce

[jira] [Created] (SPARK-38187) Resource reservation (Introduce minCPU/minMemory) with volcano implementations

2022-02-11 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38187: --- Summary: Resource reservation (Introduce minCPU/minMemory) with volcano implementations Key: SPARK-38187 URL: https://issues.apache.org/jira/browse/SPARK-38187

[jira] [Assigned] (SPARK-38184) Fix malformatted ExpressionDescription of `decode`

2022-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38184: Assignee: Xinrong Meng > Fix malformatted ExpressionDescription of `decode` >

[jira] [Resolved] (SPARK-38184) Fix malformatted ExpressionDescription of `decode`

2022-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38184. -- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-38186) Improve the README of Spark docs

2022-02-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38186. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35491

[jira] [Commented] (SPARK-38172) Adaptive coalesce not working with df persist

2022-02-11 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490773#comment-17490773 ] XiDuo You commented on SPARK-38172: --- hi [~Naveenmts]  have you tried enable this config ? {code:java}

[jira] [Resolved] (SPARK-38036) Refactor `VersionsSuite` as a subclass of `HiveVersionSuite`

2022-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38036. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35335

[jira] [Assigned] (SPARK-38036) Refactor `VersionsSuite` as a subclass of `HiveVersionSuite`

2022-02-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38036: --- Assignee: Yang Jie > Refactor `VersionsSuite` as a subclass of `HiveVersionSuite` >