[jira] [Updated] (SPARK-30211) Use python3 in make-distribution.sh

[ https://issues.apache.org/jira/browse/SPARK-30211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30211: -- Summary: Use python3 in make-distribution.sh (was: Update python version in

[jira] [Resolved] (SPARK-30211) Update python version in make-distribution.sh

[ https://issues.apache.org/jira/browse/SPARK-30211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30211. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26844

[jira] [Created] (SPARK-30214) Support COMMENT ON syntax

Kent Yao created SPARK-30214: Summary: Support COMMENT ON syntax Key: SPARK-30214 URL: https://issues.apache.org/jira/browse/SPARK-30214 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-30213) Remove the mutable status in QueryStage when enable AQE

Ke Jia created SPARK-30213: -- Summary: Remove the mutable status in QueryStage when enable AQE Key: SPARK-30213 URL: https://issues.apache.org/jira/browse/SPARK-30213 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993231#comment-16993231 ] Rinaz Belhaj commented on SPARK-19335: -- +1 This feature would be very useful. Any updates on this ?

[jira] [Updated] (SPARK-30212) COUNT(DISTINCT) window function should be supported

[ https://issues.apache.org/jira/browse/SPARK-30212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kernel Force updated SPARK-30212: - Summary: COUNT(DISTINCT) window function should be supported (was: Could not use

[jira] [Updated] (SPARK-30212) Could not use COUNT(DISTINCT) window function in SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-30212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kernel Force updated SPARK-30212: - Labels: SQL distinct window_function (was: ) > Could not use COUNT(DISTINCT) window function

[jira] [Created] (SPARK-30212) Could not use COUNT(DISTINCT) window function in SparkSQL

Dilly King created SPARK-30212: -- Summary: Could not use COUNT(DISTINCT) window function in SparkSQL Key: SPARK-30212 URL: https://issues.apache.org/jira/browse/SPARK-30212 Project: Spark Issue

[jira] [Created] (SPARK-30211) Update python version in make-distribution.sh

Yuming Wang created SPARK-30211: --- Summary: Update python version in make-distribution.sh Key: SPARK-30211 URL: https://issues.apache.org/jira/browse/SPARK-30211 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30204) Support for config Pod DNS for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vanderliang resolved SPARK-30204. - Resolution: Fixed > Support for config Pod DNS for Kubernetes >

[jira] [Assigned] (SPARK-29152) Spark Executor Plugin API shutdown is not proper when dynamic allocation enabled

[ https://issues.apache.org/jira/browse/SPARK-29152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-29152: -- Assignee: Rakesh Raushan > Spark Executor Plugin API shutdown is not

[jira] [Resolved] (SPARK-29152) Spark Executor Plugin API shutdown is not proper when dynamic allocation enabled

[ https://issues.apache.org/jira/browse/SPARK-29152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29152. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-30209) Display stageId, attemptId, taskId with SQL max metric in UI

[ https://issues.apache.org/jira/browse/SPARK-30209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993008#comment-16993008 ] Niranjan Artal commented on SPARK-30209: I am working on it. > Display stageId, attemptId,

[jira] [Updated] (SPARK-30209) Display stageId, attemptId, taskId with SQL max metric in UI

[ https://issues.apache.org/jira/browse/SPARK-30209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Artal updated SPARK-30209: --- Description: It would be helpful if we could add stageId, stage attemptId and taskId for in

[jira] [Created] (SPARK-30210) Give more informative error for BinaryClassificationEvaluator when data with only one label is provided

Paul Anzel created SPARK-30210: -- Summary: Give more informative error for BinaryClassificationEvaluator when data with only one label is provided Key: SPARK-30210 URL:

[jira] [Updated] (SPARK-30210) Give more informative error for BinaryClassificationEvaluator when data with only one label is provided

[ https://issues.apache.org/jira/browse/SPARK-30210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Anzel updated SPARK-30210: --- Description: Hi all, When I was trying to do some machine learning work with pyspark I ran into a

[jira] [Created] (SPARK-30209) Display stageId, attemptId, taskId with SQL max metric in UI

Niranjan Artal created SPARK-30209: -- Summary: Display stageId, attemptId, taskId with SQL max metric in UI Key: SPARK-30209 URL: https://issues.apache.org/jira/browse/SPARK-30209 Project: Spark

[jira] [Updated] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21869: - Fix Version/s: (was: 3.0.0) > A cached Kafka producer should not be closed if any task is

[jira] [Commented] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992975#comment-16992975 ] Shixiong Zhu commented on SPARK-21869: -- Reopened this. https://github.com/apache/spark/pull/25853

[jira] [Reopened] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-21869: -- > A cached Kafka producer should not be closed if any task is using it. >

[jira] [Resolved] (SPARK-29976) Allow speculation even if there is only one task

[ https://issues.apache.org/jira/browse/SPARK-29976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29976. --- Fix Version/s: 3.0.0 Assignee: Yuchen Huo Resolution: Fixed > Allow

[jira] [Created] (SPARK-30208) A race condition when reading from Kafka in PySpark

Shixiong Zhu created SPARK-30208: Summary: A race condition when reading from Kafka in PySpark Key: SPARK-30208 URL: https://issues.apache.org/jira/browse/SPARK-30208 Project: Spark Issue

[jira] [Resolved] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30205. --- Fix Version/s: 3.0.0 Resolution: Fixed This is resolved via 

[jira] [Assigned] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30205: - Assignee: Karthikeyan Singaravelan > Import ABC from collections.abc to remove

[jira] [Commented] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992835#comment-16992835 ] Matt Boegner commented on SPARK-30130: -- [~Ankitraj] apologies, a typo was introduced when I copied

[jira] [Updated] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Boegner updated SPARK-30130: - Description: Hardcoded numeric values in common table expressions which utilize GROUP BY are

[jira] [Assigned] (SPARK-29587) Real data type is not supported in Spark SQL which is supporting in postgresql

[ https://issues.apache.org/jira/browse/SPARK-29587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29587: --- Assignee: Kent Yao > Real data type is not supported in Spark SQL which is supporting in

[jira] [Resolved] (SPARK-29587) Real data type is not supported in Spark SQL which is supporting in postgresql

[ https://issues.apache.org/jira/browse/SPARK-29587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29587. - Fix Version/s: 3.0.0 Resolution: Fixed > Real data type is not supported in Spark SQL

[jira] [Reopened] (SPARK-29587) Real data type is not supported in Spark SQL which is supporting in postgresql

[ https://issues.apache.org/jira/browse/SPARK-29587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-29587: - > Real data type is not supported in Spark SQL which is supporting in postgresql >

[jira] [Resolved] (SPARK-30200) Add ExplainMode for Dataset.explain

[ https://issues.apache.org/jira/browse/SPARK-30200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30200. --- Fix Version/s: 3.0.0 Assignee: Takeshi Yamamuro Resolution: Fixed This is

[jira] [Created] (SPARK-30207) Enhance the SQL NULL Semantics document

Yuanjian Li created SPARK-30207: --- Summary: Enhance the SQL NULL Semantics document Key: SPARK-30207 URL: https://issues.apache.org/jira/browse/SPARK-30207 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-30125) Remove PostgreSQL dialect

[ https://issues.apache.org/jira/browse/SPARK-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30125. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26763

[jira] [Assigned] (SPARK-30125) Remove PostgreSQL dialect

[ https://issues.apache.org/jira/browse/SPARK-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30125: --- Assignee: Yuanjian Li > Remove PostgreSQL dialect > - > >

[jira] [Updated] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30205: -- Labels: (was: python3) > Import ABC from collections.abc to remove deprecation warnings >

[jira] [Updated] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30205: -- Affects Version/s: (was: 2.4.4) 3.0.0 > Import ABC from

[jira] [Updated] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30205: -- Environment: (was: Python version : 3.9 Operating System : Linux) > Import ABC from

[jira] [Updated] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30205: -- Issue Type: Improvement (was: Bug) > Import ABC from collections.abc to remove deprecation

[jira] [Updated] (SPARK-30205) Import ABC from collections.abc to remove deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-30205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30205: -- Summary: Import ABC from collections.abc to remove deprecation warnings (was: Importing ABC

[jira] [Updated] (SPARK-30197) Add minimum `requirements-dev.txt` file to `python` directory

[ https://issues.apache.org/jira/browse/SPARK-30197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30197: -- Summary: Add minimum `requirements-dev.txt` file to `python` directory (was: Add

[jira] [Closed] (SPARK-30197) Add `requirements.txt` file to `python` directory

[ https://issues.apache.org/jira/browse/SPARK-30197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-30197. - > Add `requirements.txt` file to `python` directory >

[jira] [Updated] (SPARK-30197) Add `requirements.txt` file to `python` directory

[ https://issues.apache.org/jira/browse/SPARK-30197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30197: -- Priority: Minor (was: Major) > Add `requirements.txt` file to `python` directory >

[jira] [Resolved] (SPARK-30197) Add `requirements.txt` file to `python` directory

[ https://issues.apache.org/jira/browse/SPARK-30197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30197. --- Resolution: Won't Do > Add `requirements.txt` file to `python` directory >

[jira] [Resolved] (SPARK-30206) Rename normalizeFilters in DataSourceStrategy to be generic

[ https://issues.apache.org/jira/browse/SPARK-30206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30206. --- Fix Version/s: 3.0.0 Assignee: Anton Okolnychyi Resolution: Fixed This is

[jira] [Created] (SPARK-30206) Rename normalizeFilters in DataSourceStrategy to be generic

Dongjoon Hyun created SPARK-30206: - Summary: Rename normalizeFilters in DataSourceStrategy to be generic Key: SPARK-30206 URL: https://issues.apache.org/jira/browse/SPARK-30206 Project: Spark

[jira] [Resolved] (SPARK-29967) KMeans support instance weighting

[ https://issues.apache.org/jira/browse/SPARK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29967. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26739

[jira] [Assigned] (SPARK-29967) KMeans support instance weighting

[ https://issues.apache.org/jira/browse/SPARK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29967: Assignee: Huaxin Gao > KMeans support instance weighting >

[jira] [Commented] (SPARK-20135) spark thriftserver2: no job running but containers not release on yarn

[ https://issues.apache.org/jira/browse/SPARK-20135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992644#comment-16992644 ] angerszhu commented on SPARK-20135: --- meet same problem in spark2.4.0 [~xwc3504] Have you have any

[jira] [Updated] (SPARK-30204) Support for config Pod DNS for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vanderliang updated SPARK-30204: Description: Current we can not configure the pod dns nameservers and searches when submit a job

[jira] [Updated] (SPARK-30204) Support for config Pod DNS for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vanderliang updated SPARK-30204: Description: Current we can not configure the pod dns nameservers, searches and options when

[jira] [Updated] (SPARK-30204) Support for config Pod DNS for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vanderliang updated SPARK-30204: Description: Current we can not configure the pod dns nameservers and searches when submit a job

[jira] [Created] (SPARK-30205) Importing ABC from collections module is removed in Python 3.9

Karthikeyan Singaravelan created SPARK-30205: Summary: Importing ABC from collections module is removed in Python 3.9 Key: SPARK-30205 URL: https://issues.apache.org/jira/browse/SPARK-30205

[jira] [Updated] (SPARK-30204) Support for config Pod DNS for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vanderliang updated SPARK-30204: Summary: Support for config Pod DNS for Kubernetes (was: Support for config DNS for Kubernetes)

[jira] [Created] (SPARK-30204) Support for config DNS for Kubernetes

vanderliang created SPARK-30204: --- Summary: Support for config DNS for Kubernetes Key: SPARK-30204 URL: https://issues.apache.org/jira/browse/SPARK-30204 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-30151) Issue better error message when user-specified schema not match relation schema

[ https://issues.apache.org/jira/browse/SPARK-30151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30151: --- Assignee: wuyi > Issue better error message when user-specified schema not match relation

[jira] [Resolved] (SPARK-30151) Issue better error message when user-specified schema not match relation schema

[ https://issues.apache.org/jira/browse/SPARK-30151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30151. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26781

[jira] [Updated] (SPARK-30203) store assignable if there exists an appropriate user-defined cast function

[ https://issues.apache.org/jira/browse/SPARK-30203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-30203: - Description: h3. 9.2 Store assignment h4. Syntax Rules 1) Let T be the TARGET and let V be the VALUE

[jira] [Created] (SPARK-30203) store assignable if there exists an appropriate user-defined cast function

Kent Yao created SPARK-30203: Summary: store assignable if there exists an appropriate user-defined cast function Key: SPARK-30203 URL: https://issues.apache.org/jira/browse/SPARK-30203 Project: Spark

[jira] [Created] (SPARK-30202) impl QuantileTransform

zhengruifeng created SPARK-30202: Summary: impl QuantileTransform Key: SPARK-30202 URL: https://issues.apache.org/jira/browse/SPARK-30202 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-30201) HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT

ulysses you created SPARK-30201: --- Summary: HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT Key: SPARK-30201 URL: https://issues.apache.org/jira/browse/SPARK-30201 Project:

[jira] [Commented] (SPARK-28664) ORDER BY in aggregate function

[ https://issues.apache.org/jira/browse/SPARK-28664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992427#comment-16992427 ] jiaan.geng commented on SPARK-28664:

[jira] [Commented] (SPARK-30092) Number of active tasks is negative in Live UI Executors page

[ https://issues.apache.org/jira/browse/SPARK-30092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992401#comment-16992401 ] ZhongYu commented on SPARK-30092: - It is hard to give steps that will certainly reproduce this issues.