[jira] [Updated] (SPARK-48116) Run `pyspark-pandas*` in PR builder and Daily Python CIs

[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48116: -- Summary: Run `pyspark-pandas*` in PR builder and Daily Python CIs (was: Move

[jira] [Created] (SPARK-48114) ErrorClassesJsonReader complies template regex on every template resolution

Vladimir Golubev created SPARK-48114: Summary: ErrorClassesJsonReader complies template regex on every template resolution Key: SPARK-48114 URL: https://issues.apache.org/jira/browse/SPARK-48114

[jira] [Updated] (SPARK-48115) Remove `Python 3.11` from `build_python.yml`

[ https://issues.apache.org/jira/browse/SPARK-48115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48115: --- Labels: pull-request-available (was: ) > Remove `Python 3.11` from `build_python.yml` >

[jira] [Updated] (SPARK-48114) ErrorClassesJsonReader complies template regex on every template resolution

[ https://issues.apache.org/jira/browse/SPARK-48114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48114: --- Labels: pull-request-available (was: ) > ErrorClassesJsonReader complies template regex on

[jira] [Created] (SPARK-48120) Enable autolink to SPARK jira issue

Dongjoon Hyun created SPARK-48120: - Summary: Enable autolink to SPARK jira issue Key: SPARK-48120 URL: https://issues.apache.org/jira/browse/SPARK-48120 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-45923) Spark Kubernetes Operator

[ https://issues.apache.org/jira/browse/SPARK-45923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45923: -- Affects Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Spark

[jira] [Updated] (SPARK-48120) Enable autolink to SPARK jira issue

[ https://issues.apache.org/jira/browse/SPARK-48120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48120: --- Labels: pull-request-available (was: ) > Enable autolink to SPARK jira issue >

[jira] [Resolved] (SPARK-48120) Enable autolink to SPARK jira issue

[ https://issues.apache.org/jira/browse/SPARK-48120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48120. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 11

[jira] [Updated] (SPARK-48119) Promote ` KubernetesDriverSpec` to `DeveloperApi`

[ https://issues.apache.org/jira/browse/SPARK-48119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48119: --- Labels: pull-request-available (was: ) > Promote ` KubernetesDriverSpec` to `DeveloperApi`

[jira] [Resolved] (SPARK-48118) Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable

[ https://issues.apache.org/jira/browse/SPARK-48118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48118. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46369

[jira] [Created] (SPARK-48123) Provide a constant table schema for querying structured logs

Gengliang Wang created SPARK-48123: -- Summary: Provide a constant table schema for querying structured logs Key: SPARK-48123 URL: https://issues.apache.org/jira/browse/SPARK-48123 Project: Spark

[jira] [Updated] (SPARK-48124) Disable structured logging for Interpreter by default

[ https://issues.apache.org/jira/browse/SPARK-48124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-48124: --- Description: Since there are plain text output from

[jira] [Updated] (SPARK-47336) Provide to PySpark a functionality to get estimated size of DataFrame in bytes

[ https://issues.apache.org/jira/browse/SPARK-47336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47336: --- Labels: pull-request-available (was: ) > Provide to PySpark a functionality to get

[jira] [Resolved] (SPARK-48119) Promote ` KubernetesDriverSpec` to `DeveloperApi`

[ https://issues.apache.org/jira/browse/SPARK-48119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48119. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46371

[jira] [Created] (SPARK-48121) Promote ` KubernetesDriverConf` to `DeveloperApi`

Zhou JIANG created SPARK-48121: -- Summary: Promote ` KubernetesDriverConf` to `DeveloperApi` Key: SPARK-48121 URL: https://issues.apache.org/jira/browse/SPARK-48121 Project: Spark Issue Type:

[jira] [Updated] (SPARK-48123) Provide a constant table schema for querying structured logs

[ https://issues.apache.org/jira/browse/SPARK-48123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48123: --- Labels: pull-request-available (was: ) > Provide a constant table schema for querying

[jira] [Created] (SPARK-48119) Promote ` KubernetesDriverSpec` to `DeveloperApi`

Zhou JIANG created SPARK-48119: -- Summary: Promote ` KubernetesDriverSpec` to `DeveloperApi` Key: SPARK-48119 URL: https://issues.apache.org/jira/browse/SPARK-48119 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-48121) Promote ` KubernetesDriverConf` to `DeveloperApi`

[ https://issues.apache.org/jira/browse/SPARK-48121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48121. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46373

[jira] [Updated] (SPARK-48127) Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profile` modules

[ https://issues.apache.org/jira/browse/SPARK-48127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48127: --- Labels: pull-request-available (was: ) > Fix `dev/scalastyle` to check `hadoop-cloud` and

[jira] [Updated] (SPARK-48127) Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules

[ https://issues.apache.org/jira/browse/SPARK-48127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48127: -- Summary: Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules (was: Fix

[jira] [Resolved] (SPARK-48114) ErrorClassesJsonReader complies template regex on every template resolution

[ https://issues.apache.org/jira/browse/SPARK-48114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48114. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46365

[jira] [Updated] (SPARK-48121) Promote ` KubernetesDriverConf` to `DeveloperApi`

[ https://issues.apache.org/jira/browse/SPARK-48121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48121: --- Labels: pull-request-available (was: ) > Promote ` KubernetesDriverConf` to `DeveloperApi`

[jira] [Updated] (SPARK-48124) Disable structured logging for Interpreter by default

[ https://issues.apache.org/jira/browse/SPARK-48124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-48124: --- Description: Since there are plain text output from

[jira] [Created] (SPARK-48127) Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profile` modules

Dongjoon Hyun created SPARK-48127: - Summary: Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profile` modules Key: SPARK-48127 URL: https://issues.apache.org/jira/browse/SPARK-48127 Project:

[jira] [Created] (SPARK-48118) Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable

Dongjoon Hyun created SPARK-48118: - Summary: Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable Key: SPARK-48118 URL: https://issues.apache.org/jira/browse/SPARK-48118 Project: Spark

[jira] [Resolved] (SPARK-48059) Implement the structured log framework on the java side

[ https://issues.apache.org/jira/browse/SPARK-48059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48059. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46301

[jira] [Assigned] (SPARK-48059) Implement the structured log framework on the java side

[ https://issues.apache.org/jira/browse/SPARK-48059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48059: -- Assignee: BingKun Pan > Implement the structured log framework on the java side >

[jira] [Updated] (SPARK-47484) Allow trailing comma in column definition list

[ https://issues.apache.org/jira/browse/SPARK-47484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47484: --- Labels: pull-request-available (was: ) > Allow trailing comma in column definition list >

[jira] [Created] (SPARK-48126) Make spark.log.structuredLogging.enabled effecitve

Gengliang Wang created SPARK-48126: -- Summary: Make spark.log.structuredLogging.enabled effecitve Key: SPARK-48126 URL: https://issues.apache.org/jira/browse/SPARK-48126 Project: Spark Issue

[jira] [Created] (SPARK-48124) Disable structured logging for Interpreter by default

Gengliang Wang created SPARK-48124: -- Summary: Disable structured logging for Interpreter by default Key: SPARK-48124 URL: https://issues.apache.org/jira/browse/SPARK-48124 Project: Spark

[jira] [Resolved] (SPARK-48127) Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules

[ https://issues.apache.org/jira/browse/SPARK-48127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48127. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46376

[jira] [Updated] (SPARK-48128) BitwiseCount / bit_count generated code for boolean inputs fails to compile

[ https://issues.apache.org/jira/browse/SPARK-48128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48128: --- Labels: pull-request-available (was: ) > BitwiseCount / bit_count generated code for

[jira] [Resolved] (SPARK-48116) Run `pyspark-pandas*` in PR builder and Daily Python CIs

[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48116. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46367

[jira] [Updated] (SPARK-48128) BitwiseCount / bit_count generated code for boolean inputs fails to compile

[ https://issues.apache.org/jira/browse/SPARK-48128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-48128: --- Description: If the `BitwiseCount` / `bit_count` expresison is applied to a boolean type column

[jira] [Created] (SPARK-48128) BitwiseCount / bit_count generated code for boolean inputs fails to compile

Josh Rosen created SPARK-48128: -- Summary: BitwiseCount / bit_count generated code for boolean inputs fails to compile Key: SPARK-48128 URL: https://issues.apache.org/jira/browse/SPARK-48128 Project:

[jira] [Resolved] (SPARK-48088) Skip tests being failed in client 3.5 <> server 4.0

[ https://issues.apache.org/jira/browse/SPARK-48088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48088. -- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46334

[jira] [Assigned] (SPARK-48088) Skip tests being failed in client 3.5 <> server 4.0

[ https://issues.apache.org/jira/browse/SPARK-48088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48088: Assignee: Hyukjin Kwon > Skip tests being failed in client 3.5 <> server 4.0 >

[jira] [Resolved] (SPARK-48109) Enable `k8s-integration-tests` only for `kubernetes` module change

[ https://issues.apache.org/jira/browse/SPARK-48109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48109. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46356

[jira] [Resolved] (SPARK-48103) Promote ` KubernetesDriverBuilder` to `DeveloperApi`

[ https://issues.apache.org/jira/browse/SPARK-48103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48103. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46357

[jira] [Assigned] (SPARK-48054) Backward compatibility test for Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-48054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48054: Assignee: Hyukjin Kwon > Backward compatibility test for Spark Connect >

[jira] [Resolved] (SPARK-48054) Backward compatibility test for Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-48054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48054. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46298

[jira] [Assigned] (SPARK-47240) SPIP: Structured Logging Framework for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-47240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47240: -- Assignee: (was: Apache Spark) > SPIP: Structured Logging Framework for Apache

[jira] [Assigned] (SPARK-47240) SPIP: Structured Logging Framework for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-47240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47240: -- Assignee: Apache Spark > SPIP: Structured Logging Framework for Apache Spark >

[jira] [Created] (SPARK-48112) Expose session in SparkConnectPlanner to plugin developers

Tom van Bussel created SPARK-48112: -- Summary: Expose session in SparkConnectPlanner to plugin developers Key: SPARK-48112 URL: https://issues.apache.org/jira/browse/SPARK-48112 Project: Spark

[jira] [Updated] (SPARK-48112) Expose session in SparkConnectPlanner to plugin developers

[ https://issues.apache.org/jira/browse/SPARK-48112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48112: --- Labels: pull-request-available (was: ) > Expose session in SparkConnectPlanner to plugin

[jira] [Assigned] (SPARK-47240) SPIP: Structured Logging Framework for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-47240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47240: -- Assignee: (was: Apache Spark) > SPIP: Structured Logging Framework for Apache

[jira] [Assigned] (SPARK-48112) Expose session in SparkConnectPlanner to plugin developers

[ https://issues.apache.org/jira/browse/SPARK-48112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48112: -- Assignee: Apache Spark > Expose session in SparkConnectPlanner to plugin developers

[jira] [Assigned] (SPARK-47240) SPIP: Structured Logging Framework for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-47240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47240: -- Assignee: Apache Spark > SPIP: Structured Logging Framework for Apache Spark >

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2.

[jira] [Created] (SPARK-48111) Move tpcds-1g and docker-integration-tests to daily scheduled jobs

Hyukjin Kwon created SPARK-48111: Summary: Move tpcds-1g and docker-integration-tests to daily scheduled jobs Key: SPARK-48111 URL: https://issues.apache.org/jira/browse/SPARK-48111 Project: Spark

[jira] [Resolved] (SPARK-48106) Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48106. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46353

[jira] [Assigned] (SPARK-48107) Exclude tests from Python distribution

[ https://issues.apache.org/jira/browse/SPARK-48107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48107: - Assignee: Nicholas Chammas > Exclude tests from Python distribution >

[jira] [Resolved] (SPARK-48107) Exclude tests from Python distribution

[ https://issues.apache.org/jira/browse/SPARK-48107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48107. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46354

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Attachment: (was: Screenshot 2024-05-02 at 20.59.18.png) > Reduce GitHub Action usage

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Attachment: Screenshot 2024-05-02 at 23.56.05.png > Reduce GitHub Action usage according to

[jira] [Resolved] (SPARK-48110) Remove all Maven compilation build

[ https://issues.apache.org/jira/browse/SPARK-48110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48110. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46360

[jira] [Updated] (SPARK-48111) Move tpcds-1g and docker-integration-tests to daily scheduled jobs

[ https://issues.apache.org/jira/browse/SPARK-48111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48111: --- Labels: pull-request-available (was: ) > Move tpcds-1g and docker-integration-tests to

[jira] [Resolved] (SPARK-48111) Move tpcds-1g and docker-integration-tests to daily scheduled jobs

[ https://issues.apache.org/jira/browse/SPARK-48111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48111. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46361