[jira] [Updated] (SPARK-46709) Expose partition_id column in state datasource by default

[ https://issues.apache.org/jira/browse/SPARK-46709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46709: --- Labels: pull-request-available (was: ) > Expose partition_id column in state datasource by

[jira] [Created] (SPARK-46709) Expose partition_id column in state datasource by default

Chaoqin Li created SPARK-46709: -- Summary: Expose partition_id column in state datasource by default Key: SPARK-46709 URL: https://issues.apache.org/jira/browse/SPARK-46709 Project: Spark Issue T

[jira] [Updated] (SPARK-46707) Introduce throwable field to mark expressions that may throw a runtime exception

[ https://issues.apache.org/jira/browse/SPARK-46707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46707: --- Labels: pull-request-available (was: ) > Introduce throwable field to mark expressions that

[jira] [Created] (SPARK-46708) Support error message format in Spark Connect service

Garland Zhang created SPARK-46708: - Summary: Support error message format in Spark Connect service Key: SPARK-46708 URL: https://issues.apache.org/jira/browse/SPARK-46708 Project: Spark Issue

[jira] [Resolved] (SPARK-46699) Split `ArithmeticTests`

[ https://issues.apache.org/jira/browse/SPARK-46699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46699. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44708 [https://gi

[jira] [Resolved] (SPARK-46700) count the last spilling for the shuffle disk spilling bytes metric

[ https://issues.apache.org/jira/browse/SPARK-46700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46700. --- Fix Version/s: 3.4.3 3.5.1 4.0.0 Resolution: Fix

[jira] [Assigned] (SPARK-46700) count the last spilling for the shuffle disk spilling bytes metric

[ https://issues.apache.org/jira/browse/SPARK-46700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46700: - Assignee: Wenchen Fan > count the last spilling for the shuffle disk spilling bytes met

[jira] [Updated] (SPARK-46706) percentile_approx regression since Spark 2.4

[ https://issues.apache.org/jira/browse/SPARK-46706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Pereira updated SPARK-46706: - Summary: percentile_approx regression since Spark 2.4 (was: percentile_approx regression betwee

[jira] [Updated] (SPARK-46706) percentile_approx regression between at least Spark 3.1

[ https://issues.apache.org/jira/browse/SPARK-46706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Pereira updated SPARK-46706: - Attachment: Screenshot 2024-01-05 at 3.53.10 PM.png > percentile_approx regression between at le

[jira] [Updated] (SPARK-46706) percentile_approx regression between at least Spark 3.1

[ https://issues.apache.org/jira/browse/SPARK-46706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Pereira updated SPARK-46706: - Description: Originally reported in SPARK-40499, I'm opening a new issue as that original one w

[jira] [Updated] (SPARK-46706) percentile_approx regression between at least Spark 3.1

[ https://issues.apache.org/jira/browse/SPARK-46706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Pereira updated SPARK-46706: - Description: Originally reported in SPARK-40499, I'm opening a new issue as that original one w

[jira] [Updated] (SPARK-46706) percentile_approx regression between at least Spark 3.1

[ https://issues.apache.org/jira/browse/SPARK-46706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joey Pereira updated SPARK-46706: - Attachment: Screenshot 2024-01-05 at 3.51.52 PM.png > percentile_approx regression between at le

[jira] [Created] (SPARK-46706) percentile_approx regression between at least Spark 3.1

Joey Pereira created SPARK-46706: Summary: percentile_approx regression between at least Spark 3.1 Key: SPARK-46706 URL: https://issues.apache.org/jira/browse/SPARK-46706 Project: Spark Issue

[jira] [Commented] (SPARK-40499) Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-40499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806201#comment-17806201 ] Joey Pereira commented on SPARK-40499: -- I've found another app of ours with similar

[jira] [Commented] (SPARK-40499) Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-40499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806202#comment-17806202 ] Joey Pereira commented on SPARK-40499: -- (I realized I can't really re-open the tick

[jira] [Resolved] (SPARK-46704) Fix `MasterPage` to sort `Running Drivers` table by `Duration` column correctly

[ https://issues.apache.org/jira/browse/SPARK-46704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46704. --- Fix Version/s: 3.4.3 3.5.1 4.0.0 Resolution: Fix

[jira] [Resolved] (SPARK-46703) Disable `fail-fast` in Python CI

[ https://issues.apache.org/jira/browse/SPARK-46703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46703. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44710 [https://

[jira] [Updated] (SPARK-46704) Fix `MasterPage` to sort `Running Drivers` table by `Duration` column correctly

[ https://issues.apache.org/jira/browse/SPARK-46704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46704: -- Affects Version/s: 3.3.4 3.5.0 3.4.1

[jira] [Updated] (SPARK-46703) Disable `fail-fast` in Python CI

[ https://issues.apache.org/jira/browse/SPARK-46703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46703: --- Labels: pull-request-available (was: ) > Disable `fail-fast` in Python CI > --

[jira] [Created] (SPARK-46703) Disable `fail-fast` in Python CI

Dongjoon Hyun created SPARK-46703: - Summary: Disable `fail-fast` in Python CI Key: SPARK-46703 URL: https://issues.apache.org/jira/browse/SPARK-46703 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Labels: databricks (was: ) > Spark Cluster Crashing > -- > >

[jira] [Commented] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806156#comment-17806156 ] Mohamad Haidar commented on SPARK-46702: [~garren], can you please help on that,

[jira] [Commented] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806150#comment-17806150 ] Nicholas Chammas commented on SPARK-45599: -- cc [~dongjoon] - This is an old cor

[jira] [Updated] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-45599: - Labels: correctness (was: data-corruption) > Percentile can produce a wrong answer if -

[jira] [Commented] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806148#comment-17806148 ] Nicholas Chammas commented on SPARK-45599: -- I can confirm that this bug is stil

[jira] [Updated] (SPARK-46395) Assign Spark configs to groups for use in documentation

[ https://issues.apache.org/jira/browse/SPARK-46395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-46395: - Summary: Assign Spark configs to groups for use in documentation (was: Automatically ge

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Description: h3. Description: * We have a spark cluster installed over a k8s cluster with o

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: image-2024-01-12-10-45-30-398.png > Spark Cluster Crashing > ---

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: image-2024-01-12-10-45-18-905.png > Spark Cluster Crashing > ---

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Description: h3. Description: h3. 1.   We have a spark cluster installed over a k8s clus

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: image-2024-01-12-10-45-50-427.png > Spark Cluster Crashing > ---

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: cveshv-events-streaming-TRACE (2).zip > Spark Cluster Crashing > ---

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: image-2024-01-12-10-45-40-397.png > Spark Cluster Crashing > ---

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: image-2024-01-12-10-44-45-717.png > Spark Cluster Crashing > ---

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: cveshv-events-streaming-core-cp-type2-filter-driver-0_zrdm71bnrt201_1201240818.l

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Description: h3. Description: h3. 1.   We have a spark cluster installed over a k8s clus

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: CV62A4~1.LOG > Spark Cluster Crashing > -- > >

[jira] [Updated] (SPARK-46702) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohamad Haidar updated SPARK-46702: --- Attachment: logs_cveshv-events-streaming-core-cp-type2-filter-driver-0_051023_1500_prev.log

[jira] [Created] (SPARK-46702) Spark Cluster Crashing

Mohamad Haidar created SPARK-46702: -- Summary: Spark Cluster Crashing Key: SPARK-46702 URL: https://issues.apache.org/jira/browse/SPARK-46702 Project: Spark Issue Type: Bug Componen

[jira] [Updated] (SPARK-46701) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hussein Ballout updated SPARK-46701: Environment: Apache Spark 3.4.0     was: Kubernetes: 1.27 Apache Spark 3.4.0   > S

[jira] [Created] (SPARK-46701) Spark Cluster Crashing

Hussein Ballout created SPARK-46701: --- Summary: Spark Cluster Crashing Key: SPARK-46701 URL: https://issues.apache.org/jira/browse/SPARK-46701 Project: Spark Issue Type: Bug Compo

[jira] [Updated] (SPARK-46701) Spark Cluster Crashing

[ https://issues.apache.org/jira/browse/SPARK-46701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hussein Ballout updated SPARK-46701: Shepherd: Mohamad Haidar (was: Garren Smith) > Spark Cluster Crashing >

[jira] [Updated] (SPARK-46700) count the last spilling for the shuffle disk spilling bytes metric

[ https://issues.apache.org/jira/browse/SPARK-46700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46700: --- Labels: pull-request-available (was: ) > count the last spilling for the shuffle disk spill

[jira] [Created] (SPARK-46700) count the last spilling for the shuffle disk spilling bytes metric

Wenchen Fan created SPARK-46700: --- Summary: count the last spilling for the shuffle disk spilling bytes metric Key: SPARK-46700 URL: https://issues.apache.org/jira/browse/SPARK-46700 Project: Spark

[jira] [Updated] (SPARK-46699) Split `ArithmeticTests`

[ https://issues.apache.org/jira/browse/SPARK-46699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46699: --- Labels: pull-request-available (was: ) > Split `ArithmeticTests` > ---

[jira] [Created] (SPARK-46699) Split `ArithmeticTests`

Ruifeng Zheng created SPARK-46699: - Summary: Split `ArithmeticTests` Key: SPARK-46699 URL: https://issues.apache.org/jira/browse/SPARK-46699 Project: Spark Issue Type: Sub-task Comp

[jira] [Updated] (SPARK-27950) Additional DynamoDB and CloudWatch config

[ https://issues.apache.org/jira/browse/SPARK-27950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-27950: --- Labels: pull-request-available (was: ) > Additional DynamoDB and CloudWatch config > --

[jira] [Resolved] (SPARK-46695) Always setting hive.execution.engine to mr

[ https://issues.apache.org/jira/browse/SPARK-46695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46695. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44704 [https://

[jira] [Resolved] (SPARK-46698) Replace Timer with single thread scheduled executor for ConsoleProgressBar

[ https://issues.apache.org/jira/browse/SPARK-46698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46698. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44701 [https://

[jira] [Assigned] (SPARK-46698) Replace Timer with single thread scheduled executor for ConsoleProgressBar

[ https://issues.apache.org/jira/browse/SPARK-46698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46698: - Assignee: Jiaan Geng (was: Yang Jie) > Replace Timer with single thread scheduled exec

[jira] [Assigned] (SPARK-46698) Replace Timer with single thread scheduled executor for ConsoleProgressBar

[ https://issues.apache.org/jira/browse/SPARK-46698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46698: - Assignee: Yang Jie > Replace Timer with single thread scheduled executor for ConsolePro

[jira] [Resolved] (SPARK-46681) Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured

[ https://issues.apache.org/jira/browse/SPARK-46681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46681. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44691 [https://

[jira] [Assigned] (SPARK-46681) Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured

[ https://issues.apache.org/jira/browse/SPARK-46681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46681: - Assignee: Yang Jie > Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid

[jira] [Assigned] (SPARK-46692) Fix potential environmental variable transfer issue in `pyspark job GA`

[ https://issues.apache.org/jira/browse/SPARK-46692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46692: - Assignee: BingKun Pan > Fix potential environmental variable transfer issue in `pyspark

[jira] [Resolved] (SPARK-46692) Fix potential environmental variable transfer issue in `pyspark job GA`

[ https://issues.apache.org/jira/browse/SPARK-46692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46692. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44698 [https://

[jira] [Updated] (SPARK-46692) Load `inputs.envs` always in `Python` and `Upload`-related steps

[ https://issues.apache.org/jira/browse/SPARK-46692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46692: -- Summary: Load `inputs.envs` always in `Python` and `Upload`-related steps (was: Fix potential

[jira] [Updated] (SPARK-46692) Fix potential environmental variable transfer issue in `pyspark job GA`

[ https://issues.apache.org/jira/browse/SPARK-46692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-46692: Summary: Fix potential environmental variable transfer issue in `pyspark job GA` (was: Fix potent

[jira] [Resolved] (SPARK-46665) Remove assertPandasOnSparkEqual

[ https://issues.apache.org/jira/browse/SPARK-46665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46665. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44675 [https://

[jira] [Assigned] (SPARK-46665) Remove assertPandasOnSparkEqual

[ https://issues.apache.org/jira/browse/SPARK-46665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46665: - Assignee: Haejoon Lee > Remove assertPandasOnSparkEqual > -

[jira] [Updated] (SPARK-46698) Replace Timer with single thread scheduled executor for ConsoleProgressBar

[ https://issues.apache.org/jira/browse/SPARK-46698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46698: --- Labels: pull-request-available (was: ) > Replace Timer with single thread scheduled executo

[jira] [Created] (SPARK-46698) Replace Timer with single thread scheduled executor for ConsoleProgressBar

Yang Jie created SPARK-46698: Summary: Replace Timer with single thread scheduled executor for ConsoleProgressBar Key: SPARK-46698 URL: https://issues.apache.org/jira/browse/SPARK-46698 Project: Spark

[jira] [Assigned] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

[ https://issues.apache.org/jira/browse/SPARK-46052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-46052: --- Assignee: wuyi > Remove unnecessary TaskScheduler.killAllTaskAttempts > ---

[jira] [Resolved] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

[ https://issues.apache.org/jira/browse/SPARK-46052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46052. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43954 [https://gith

[jira] [Assigned] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

[ https://issues.apache.org/jira/browse/SPARK-46052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46052: -- Assignee: (was: Apache Spark) > Remove unnecessary TaskScheduler.killAllTaskAttem

[jira] [Assigned] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

[ https://issues.apache.org/jira/browse/SPARK-46052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46052: -- Assignee: Apache Spark > Remove unnecessary TaskScheduler.killAllTaskAttempts > -