[jira] [Created] (SPARK-48362) Add CollectSetWIthLimit

2024-05-20 Thread Holden Karau (Jira)
Holden Karau created SPARK-48362: Summary: Add CollectSetWIthLimit Key: SPARK-48362 URL: https://issues.apache.org/jira/browse/SPARK-48362 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2024-05-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-44953: Assignee: binjie yang > Log a warning (or automatically disable) when shuffle tracking

[jira] [Resolved] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2024-05-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-44953. -- Resolution: Fixed > Log a warning (or automatically disable) when shuffle tracking is enabled

[jira] [Updated] (SPARK-48101) When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved

2024-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-48101: - Priority: Minor (was: Major) > When using INSERT OVERWRITE with Spark CTEs they may not be

[jira] [Created] (SPARK-48101) When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved

2024-05-02 Thread Holden Karau (Jira)
Holden Karau created SPARK-48101: Summary: When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved Key: SPARK-48101 URL: https://issues.apache.org/jira/browse/SPARK-48101 Project:

[jira] [Created] (SPARK-47672) Avoid double evaluation of non-trivial projected elements from filter pushdown

2024-04-01 Thread Holden Karau (Jira)
Holden Karau created SPARK-47672: Summary: Avoid double evaluation of non-trivial projected elements from filter pushdown Key: SPARK-47672 URL: https://issues.apache.org/jira/browse/SPARK-47672

[jira] [Created] (SPARK-47220) log4j race condition during shutdown

2024-02-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-47220: Summary: log4j race condition during shutdown Key: SPARK-47220 URL: https://issues.apache.org/jira/browse/SPARK-47220 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-47077) sbt build is broken due to selenium change

2024-02-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-47077. -- Resolution: Cannot Reproduce After blowing away my maven + ivy cache it works fine – should

[jira] [Created] (SPARK-47077) sbt build is broken due to selenium change

2024-02-16 Thread Holden Karau (Jira)
Holden Karau created SPARK-47077: Summary: sbt build is broken due to selenium change Key: SPARK-47077 URL: https://issues.apache.org/jira/browse/SPARK-47077 Project: Spark Issue Type:

[jira] [Updated] (SPARK-47001) Pushdown Verification in Optimizer.scala should support changed data types

2024-02-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-47001: - Description: When pushing a filter down in a union the data type may not match exactly if the

[jira] [Created] (SPARK-47031) Union of with non-determinstic expression should be non-deterministic

2024-02-12 Thread Holden Karau (Jira)
Holden Karau created SPARK-47031: Summary: Union of with non-determinstic expression should be non-deterministic Key: SPARK-47031 URL: https://issues.apache.org/jira/browse/SPARK-47031 Project: Spark

[jira] [Created] (SPARK-45712) Provide a command line flag to override the log4j properties file

2023-10-27 Thread Holden Karau (Jira)
Holden Karau created SPARK-45712: Summary: Provide a command line flag to override the log4j properties file Key: SPARK-45712 URL: https://issues.apache.org/jira/browse/SPARK-45712 Project: Spark

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Affects Version/s: 4.0.0 (was: 3.3.0)

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Issue Type: Improvement (was: Bug) > Spark history files backend currently depend on polling

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Description: The spark history server FS  currently depends on polling for loading history

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Summary: Spark history files backend currently depend on polling for loading into the history

[jira] [Created] (SPARK-45563) Spark rolling history files currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
Holden Karau created SPARK-45563: Summary: Spark rolling history files currently depend on polling for loading into the history server Key: SPARK-45563 URL: https://issues.apache.org/jira/browse/SPARK-45563

[jira] [Resolved] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-10-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-44735. -- Fix Version/s: 4.0.0 Resolution: Fixed > Log a warning when inserting columns with the

[jira] [Assigned] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-10-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-44735: Assignee: Jia Fan > Log a warning when inserting columns with the same name by row that

[jira] [Created] (SPARK-44992) Add support for rack information from an environment variable

2023-08-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-44992: Summary: Add support for rack information from an environment variable Key: SPARK-44992 URL: https://issues.apache.org/jira/browse/SPARK-44992 Project: Spark

[jira] [Created] (SPARK-44970) Spark History File Uploads Can Fail on S3

2023-08-25 Thread Holden Karau (Jira)
Holden Karau created SPARK-44970: Summary: Spark History File Uploads Can Fail on S3 Key: SPARK-44970 URL: https://issues.apache.org/jira/browse/SPARK-44970 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-44955) Add the option for dynamically marking containers for preemption based data

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44955: Summary: Add the option for dynamically marking containers for preemption based data Key: SPARK-44955 URL: https://issues.apache.org/jira/browse/SPARK-44955 Project:

[jira] [Created] (SPARK-44954) Make DEA algorithms pluggable

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44954: Summary: Make DEA algorithms pluggable Key: SPARK-44954 URL: https://issues.apache.org/jira/browse/SPARK-44954 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2023-08-24 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-44953: - Parent: SPARK-44951 Issue Type: Sub-task (was: Improvement) > Log a warning (or

[jira] [Created] (SPARK-44953) Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44953: Summary: Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism Key: SPARK-44953 URL:

[jira] [Created] (SPARK-44951) Improve Spark Dynamic Allocation

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44951: Summary: Improve Spark Dynamic Allocation Key: SPARK-44951 URL: https://issues.apache.org/jira/browse/SPARK-44951 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-44950) Improve Spark Driver Launch Time

2023-08-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44950: Summary: Improve Spark Driver Launch Time Key: SPARK-44950 URL: https://issues.apache.org/jira/browse/SPARK-44950 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-44769) Add SQL statement to create an empty array with a type

2023-08-10 Thread Holden Karau (Jira)
Holden Karau created SPARK-44769: Summary: Add SQL statement to create an empty array with a type Key: SPARK-44769 URL: https://issues.apache.org/jira/browse/SPARK-44769 Project: Spark Issue

[jira] [Updated] (SPARK-42035) Add a config flag to force exit on JDK major version mismatch

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42035: - Target Version/s: 4.0.0 > Add a config flag to force exit on JDK major version mismatch >

[jira] [Updated] (SPARK-42261) K8s will not allocate more execs if there are any pending execs until next snapshot

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42261: - Target Version/s: 4.0.0 > K8s will not allocate more execs if there are any pending execs until

[jira] [Updated] (SPARK-44511) Allow insertInto to succeed with partion columns specified when they match those on the target table

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-44511: - Target Version/s: 4.0.0 > Allow insertInto to succeed with partion columns specified when they

[jira] [Updated] (SPARK-42361) Add an option to use external storage to distribute JAR set in cluster mode on Kube

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42361: - Target Version/s: 4.0.0 > Add an option to use external storage to distribute JAR set in

[jira] [Updated] (SPARK-42260) Log when the K8s Exec Pods Allocator Stalls

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42260: - Target Version/s: 4.0.0 > Log when the K8s Exec Pods Allocator Stalls >

[jira] [Commented] (SPARK-44727) Improve the error message for dynamic allocation conditions

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752496#comment-17752496 ] Holden Karau commented on SPARK-44727: -- Do you have more context [~chengpan] ? > Improve the error

[jira] [Updated] (SPARK-42035) Add a config flag to force exit on JDK major version mismatch

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-42035: - Description: JRE version mismatches can cause errors which are difficult to debug (potentially

[jira] [Updated] (SPARK-34337) Reject disk blocks when out of disk space

2023-08-09 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34337: - Target Version/s: 4.0.0 > Reject disk blocks when out of disk space >

[jira] [Created] (SPARK-44735) Log a warning when inserting columns with the same name by row that don't match up

2023-08-08 Thread Holden Karau (Jira)
Holden Karau created SPARK-44735: Summary: Log a warning when inserting columns with the same name by row that don't match up Key: SPARK-44735 URL: https://issues.apache.org/jira/browse/SPARK-44735

[jira] [Commented] (SPARK-44050) Unable to Mount ConfigMap in Driver Pod - ConfigMap Creation Issue

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751796#comment-17751796 ] Holden Karau commented on SPARK-44050: -- Ah interesting, it sounds like the fix would be to retry

[jira] [Commented] (SPARK-44508) Add user guide for Python UDTFs

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751795#comment-17751795 ] Holden Karau commented on SPARK-44508: -- I'm not sure this should be a blocker. > Add user guide

[jira] [Commented] (SPARK-24282) Add support for PMML export for the Standard Scaler Stage

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751758#comment-17751758 ] Holden Karau commented on SPARK-24282: -- I don't think were going to do this anymore. > Add support

[jira] [Resolved] (SPARK-28740) Add support for building with bloop

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-28740. -- Resolution: Won't Fix > Add support for building with bloop >

[jira] [Commented] (SPARK-32111) Cleanup locks and docs in CoarseGrainedSchedulerBackend

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751754#comment-17751754 ] Holden Karau commented on SPARK-32111: -- I think this could be a good target for Spark 4.0 >

[jira] [Updated] (SPARK-32111) Cleanup locks and docs in CoarseGrainedSchedulerBackend

2023-08-07 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-32111: - Target Version/s: 4.0.0 Affects Version/s: 4.0.0 > Cleanup locks and docs in

[jira] [Updated] (SPARK-44578) Support pushing down BoundFunction in DSv2

2023-07-31 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-44578: - Description: See [https://github.com/apache/iceberg/pull/7886#discussion_r1257537662]  (was:

[jira] [Created] (SPARK-44592) Add history link to previous run on SS HA

2023-07-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-44592: Summary: Add history link to previous run on SS HA Key: SPARK-44592 URL: https://issues.apache.org/jira/browse/SPARK-44592 Project: Spark Issue Type:

[jira] [Created] (SPARK-44578) Support pushing down UDFs in DSv2

2023-07-27 Thread Holden Karau (Jira)
Holden Karau created SPARK-44578: Summary: Support pushing down UDFs in DSv2 Key: SPARK-44578 URL: https://issues.apache.org/jira/browse/SPARK-44578 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

2023-07-26 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747676#comment-17747676 ] Holden Karau commented on SPARK-37562: -- So (in theory) the cluster administrator has some base

[jira] [Created] (SPARK-44529) Add a flag to resolve docker tags to hashes at launch time

2023-07-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-44529: Summary: Add a flag to resolve docker tags to hashes at launch time Key: SPARK-44529 URL: https://issues.apache.org/jira/browse/SPARK-44529 Project: Spark

[jira] [Created] (SPARK-44511) Allow insertInto to succeed with partion columns specified when they match those on the target table

2023-07-21 Thread Holden Karau (Jira)
Holden Karau created SPARK-44511: Summary: Allow insertInto to succeed with partion columns specified when they match those on the target table Key: SPARK-44511 URL:

[jira] [Created] (SPARK-42762) Improve Logging for disconnects during exec id request

2023-03-11 Thread Holden Karau (Jira)
Holden Karau created SPARK-42762: Summary: Improve Logging for disconnects during exec id request Key: SPARK-42762 URL: https://issues.apache.org/jira/browse/SPARK-42762 Project: Spark Issue

[jira] [Commented] (SPARK-42411) Better support for Istio service mesh while running Spark on Kubernetes

2023-02-12 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687664#comment-17687664 ] Holden Karau commented on SPARK-42411: -- The other option is something around

[jira] [Created] (SPARK-42361) Add an option to use external storage to distribute JAR set in cluster mode on Kube

2023-02-06 Thread Holden Karau (Jira)
Holden Karau created SPARK-42361: Summary: Add an option to use external storage to distribute JAR set in cluster mode on Kube Key: SPARK-42361 URL: https://issues.apache.org/jira/browse/SPARK-42361

[jira] [Created] (SPARK-42261) K8s will not allocate more execs if there are any pending execs until next snapshot

2023-01-31 Thread Holden Karau (Jira)
Holden Karau created SPARK-42261: Summary: K8s will not allocate more execs if there are any pending execs until next snapshot Key: SPARK-42261 URL: https://issues.apache.org/jira/browse/SPARK-42261

[jira] [Created] (SPARK-42260) Log when the K8s Exec Pods Allocator Stalls

2023-01-31 Thread Holden Karau (Jira)
Holden Karau created SPARK-42260: Summary: Log when the K8s Exec Pods Allocator Stalls Key: SPARK-42260 URL: https://issues.apache.org/jira/browse/SPARK-42260 Project: Spark Issue Type:

[jira] [Created] (SPARK-42173) IPv6 address mapping can fail with sparse addresses

2023-01-24 Thread Holden Karau (Jira)
Holden Karau created SPARK-42173: Summary: IPv6 address mapping can fail with sparse addresses Key: SPARK-42173 URL: https://issues.apache.org/jira/browse/SPARK-42173 Project: Spark Issue

[jira] [Created] (SPARK-42035) Add a config flag to force exit on JDK major version mismatch

2023-01-12 Thread Holden Karau (Jira)
Holden Karau created SPARK-42035: Summary: Add a config flag to force exit on JDK major version mismatch Key: SPARK-42035 URL: https://issues.apache.org/jira/browse/SPARK-42035 Project: Spark

[jira] [Resolved] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2022-12-12 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33782. -- Fix Version/s: 3.4.0 Assignee: Pralabh Kumar Resolution: Fixed > Place

[jira] [Resolved] (SPARK-39799) DataSourceV2: View catalog interface

2022-11-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-39799. -- Fix Version/s: 3.4.0 Resolution: Fixed > DataSourceV2: View catalog interface >

[jira] [Assigned] (SPARK-39799) DataSourceV2: View catalog interface

2022-11-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-39799: Assignee: John Zhuge > DataSourceV2: View catalog interface >

[jira] [Created] (SPARK-40850) Tests for Spark SQL Intrepetered Queries may execute Codegen

2022-10-19 Thread Holden Karau (Jira)
Holden Karau created SPARK-40850: Summary: Tests for Spark SQL Intrepetered Queries may execute Codegen Key: SPARK-40850 URL: https://issues.apache.org/jira/browse/SPARK-40850 Project: Spark

[jira] [Resolved] (SPARK-40428) Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown

2022-10-04 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-40428. -- Fix Version/s: 3.4.0 Assignee: Holden Karau Resolution: Fixed > Add a

[jira] [Created] (SPARK-40428) Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown

2022-09-14 Thread Holden Karau (Jira)
Holden Karau created SPARK-40428: Summary: Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown Key: SPARK-40428 URL:

[jira] [Created] (SPARK-40379) Propagate decommission executor loss reason during onDisconnect in K8s

2022-09-07 Thread Holden Karau (Jira)
Holden Karau created SPARK-40379: Summary: Propagate decommission executor loss reason during onDisconnect in K8s Key: SPARK-40379 URL: https://issues.apache.org/jira/browse/SPARK-40379 Project:

[jira] [Resolved] (SPARK-38969) Graceful decomissionning on Kubernetes fails / decom script error

2022-08-12 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-38969. -- Fix Version/s: 3.4.0 Assignee: Holden Karau Resolution: Fixed Updated

[jira] [Created] (SPARK-39512) Document the Spark Docker container release process

2022-06-18 Thread Holden Karau (Jira)
Holden Karau created SPARK-39512: Summary: Document the Spark Docker container release process Key: SPARK-39512 URL: https://issues.apache.org/jira/browse/SPARK-39512 Project: Spark Issue

[jira] [Created] (SPARK-39088) Add support for forwarding Spark History requests to live running driver when present

2022-05-02 Thread Holden Karau (Jira)
Holden Karau created SPARK-39088: Summary: Add support for forwarding Spark History requests to live running driver when present Key: SPARK-39088 URL: https://issues.apache.org/jira/browse/SPARK-39088

[jira] [Commented] (SPARK-38969) Graceful decomissionning on Kubernetes fails / decom script error

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530922#comment-17530922 ] Holden Karau commented on SPARK-38969: -- Most likely it's from the `timeout 60` in there (at least

[jira] [Commented] (SPARK-38158) Rdd and shuffle blocks not migrated to new executors when decommission feature is enabled

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530916#comment-17530916 ] Holden Karau commented on SPARK-38158: -- Do you have executor logs? > Rdd and shuffle blocks not

[jira] [Commented] (SPARK-38408) Add option for partial nested field writes with nullable fields.

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530915#comment-17530915 ] Holden Karau commented on SPARK-38408: -- More general > Add option for partial nested field writes

[jira] [Resolved] (SPARK-33178) Dynamic Scaling In/Scaling out of executors - Kubernetes

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33178. -- Resolution: Incomplete Resolved since there is not enough info for what the request is for.

[jira] [Resolved] (SPARK-34104) Allow users to specify a maximum decommissioning time

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34104. -- Fix Version/s: 3.2.0 Resolution: Fixed > Allow users to specify a maximum

[jira] [Resolved] (SPARK-34105) In addition to killing exlcuded/flakey executors which should support decommissioning

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34105. -- Fix Version/s: 3.2.0 Resolution: Fixed > In addition to killing exlcuded/flakey

[jira] [Resolved] (SPARK-34530) logError for interrupting block migrations is too high

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34530. -- Resolution: Incomplete > logError for interrupting block migrations is too high >

[jira] [Resolved] (SPARK-36664) Log time spent waiting for cluster resources

2022-05-02 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-36664. -- Fix Version/s: 3.4.0 Assignee: Holden Karau Resolution: Fixed > Log time

[jira] [Resolved] (SPARK-38561) Add doc for "Customized Kubernetes Schedulers"

2022-03-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-38561. -- Fix Version/s: 3.3.0 Assignee: Yikun Jiang Resolution: Fixed > Add doc for

[jira] [Updated] (SPARK-38503) Add warn for getAdditionalPreKubernetesResources in executor side

2022-03-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38503: - Target Version/s: 3.4.0 > Add warn for getAdditionalPreKubernetesResources in executor side >

[jira] [Created] (SPARK-38408) Add option for partial nested field writes with nullable fields.

2022-03-03 Thread Holden Karau (Jira)
Holden Karau created SPARK-38408: Summary: Add option for partial nested field writes with nullable fields. Key: SPARK-38408 URL: https://issues.apache.org/jira/browse/SPARK-38408 Project: Spark

[jira] [Created] (HADOOP-18132) S3 exponential backoff

2022-02-18 Thread Holden Karau (Jira)
Holden Karau created HADOOP-18132: - Summary: S3 exponential backoff Key: HADOOP-18132 URL: https://issues.apache.org/jira/browse/HADOOP-18132 Project: Hadoop Common Issue Type: Improvement

[jira] [Created] (HADOOP-18132) S3 exponential backoff

2022-02-18 Thread Holden Karau (Jira)
Holden Karau created HADOOP-18132: - Summary: S3 exponential backoff Key: HADOOP-18132 URL: https://issues.apache.org/jira/browse/HADOOP-18132 Project: Hadoop Common Issue Type: Improvement

[jira] [Created] (SPARK-38090) Make links for stderr/stdout on Spark on Kube configurable

2022-02-02 Thread Holden Karau (Jira)
Holden Karau created SPARK-38090: Summary: Make links for stderr/stdout on Spark on Kube configurable Key: SPARK-38090 URL: https://issues.apache.org/jira/browse/SPARK-38090 Project: Spark

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-02-01 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38075: - Fix Version/s: 3.1.3 (was: 3.1.4) > Hive script transform with order by

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-02-01 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38075: - Fix Version/s: 3.1.4 (was: 3.1.3) Affects Version/s: 3.1.3 >

[jira] [Updated] (SPARK-36782) Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster upon migrating shuffle blocks

2022-01-10 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-36782: - Fix Version/s: 3.1.3 > Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster

[jira] [Commented] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

2021-12-06 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454194#comment-17454194 ] Holden Karau commented on SPARK-37562: -- cc [~dongjoon] any thoughts on the param name we should

[jira] [Created] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

2021-12-06 Thread Holden Karau (Jira)
Holden Karau created SPARK-37562: Summary: Add Spark History Server Links for Kubernetes & other CMs Key: SPARK-37562 URL: https://issues.apache.org/jira/browse/SPARK-37562 Project: Spark

[jira] [Updated] (SPARK-37359) Spark K8s tests are broken

2021-11-17 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-37359: - Description: For decom: We should look for "Decommission executors" instead of "Remove reason

[jira] [Updated] (SPARK-37359) Spark K8s tests are broken

2021-11-17 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-37359: - Summary: Spark K8s tests are broken (was: Spark K8s decom test looks for a non longer current

[jira] [Created] (SPARK-37359) Spark K8s decom test looks for a non longer current string

2021-11-17 Thread Holden Karau (Jira)
Holden Karau created SPARK-37359: Summary: Spark K8s decom test looks for a non longer current string Key: SPARK-37359 URL: https://issues.apache.org/jira/browse/SPARK-37359 Project: Spark

[jira] [Updated] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-33782: - Target Version/s: 3.3.0 > Place spark.files, spark.jars and spark.files under the current

[jira] [Commented] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414624#comment-17414624 ] Holden Karau commented on SPARK-33782: -- I think this missed the window for Spark 3.2, but I'm happy

[jira] [Resolved] (SPARK-33885) The position of unresolved identifier for DDL commands should be respected..

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33885. -- Fix Version/s: 3.2.0 Assignee: Terry Kim Resolution: Fixed > The position of

[jira] [Commented] (SPARK-34019) Keep same quantiles of UI and restful API

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414622#comment-17414622 ] Holden Karau commented on SPARK-34019: -- This is targeting 4 since it's backwards incompat change.

[jira] [Commented] (SPARK-34064) Broadcast job is not aborted even the SQL statement canceled

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414621#comment-17414621 ] Holden Karau commented on SPARK-34064: -- [~inetfuture]it's hard to say since the initial fix was

[jira] [Resolved] (SPARK-34156) Unify the output of DDL and pass output attributes properly

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-34156. -- Fix Version/s: 3.2.0 Resolution: Fixed > Unify the output of DDL and pass output

[jira] [Commented] (SPARK-34208) Upgrade ORC to 1.6.7

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414618#comment-17414618 ] Holden Karau commented on SPARK-34208: -- Is ORC-965 a regression and if so should we switch this to

[jira] [Commented] (SPARK-34156) Unify the output of DDL and pass output attributes properly

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414619#comment-17414619 ] Holden Karau commented on SPARK-34156: -- All of the sub issues are resolved so I'm going to go ahead

[jira] [Commented] (SPARK-34329) When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414617#comment-17414617 ] Holden Karau commented on SPARK-34329: -- Is this a regresion or has this behaviour been around in

[jira] [Updated] (SPARK-34208) Upgrade ORC to 1.6.7

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34208: - Description: Apache ORC 1.6.7 has the following fixes including ORC-711 Support

[jira] [Updated] (SPARK-34478) Ignore or reject wrong config when start sparksession

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34478: - Priority: Minor (was: Trivial) > Ignore or reject wrong config when start sparksession >

[jira] [Updated] (SPARK-34478) Ignore or reject wrong config when start sparksession

2021-09-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-34478: - Priority: Trivial (was: Major) > Ignore or reject wrong config when start sparksession >

  1   2   3   4   5   >