[jira] [Resolved] (SPARK-48207) Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48207. --- Fix Version/s: 3.4.4 Resolution: Fixed Issue resolved by pull request 46489 [https://github.com/apache/spark/pull/46489] > Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed > > > Key: SPARK-48207 > URL: https://issues.apache.org/jira/browse/SPARK-48207 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.4 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48207) Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48207: - Assignee: Dongjoon Hyun > Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed > > > Key: SPARK-48207 > URL: https://issues.apache.org/jira/browse/SPARK-48207 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.4 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48207) Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48207: -- Summary: Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed (was: Run build/scala-213/java-11-17 jobs of `branch-3.4` only if needed) > Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed > > > Key: SPARK-48207 > URL: https://issues.apache.org/jira/browse/SPARK-48207 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.4 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48207) Run build/scala-213/java-11-17 jobs of `branch-3.4` only if needed
Dongjoon Hyun created SPARK-48207: - Summary: Run build/scala-213/java-11-17 jobs of `branch-3.4` only if needed Key: SPARK-48207 URL: https://issues.apache.org/jira/browse/SPARK-48207 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 3.4.4 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48192) Enable TPC-DS and docker tests in forked repository
[ https://issues.apache.org/jira/browse/SPARK-48192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48192: -- Fix Version/s: 3.4.4 > Enable TPC-DS and docker tests in forked repository > --- > > Key: SPARK-48192 > URL: https://issues.apache.org/jira/browse/SPARK-48192 > Project: Spark > Issue Type: Sub-task > Components: Project Infra, SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > > TPC-DS is pretty important in SQL. Shoud at least enable it in forked > repositories (PR builders) which does not consume ASF resource. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48132) Run `k8s-integration-tests` only in PR builder and Daily CIs
[ https://issues.apache.org/jira/browse/SPARK-48132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48132: -- Fix Version/s: 3.4.4 > Run `k8s-integration-tests` only in PR builder and Daily CIs > > > Key: SPARK-48132 > URL: https://issues.apache.org/jira/browse/SPARK-48132 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48192) Enable TPC-DS and docker tests in forked repository
[ https://issues.apache.org/jira/browse/SPARK-48192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48192: -- Summary: Enable TPC-DS and docker tests in forked repository (was: Enable TPC-DS tests in forked repository) > Enable TPC-DS and docker tests in forked repository > --- > > Key: SPARK-48192 > URL: https://issues.apache.org/jira/browse/SPARK-48192 > Project: Spark > Issue Type: Sub-task > Components: Project Infra, SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > TPC-DS is pretty important in SQL. Shoud at least enable it in forked > repositories (PR builders) which does not consume ASF resource. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48192) Enable TPC-DS tests in forked repository
[ https://issues.apache.org/jira/browse/SPARK-48192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48192: -- Fix Version/s: 3.5.2 > Enable TPC-DS tests in forked repository > > > Key: SPARK-48192 > URL: https://issues.apache.org/jira/browse/SPARK-48192 > Project: Spark > Issue Type: Sub-task > Components: Project Infra, SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > TPC-DS is pretty important in SQL. Shoud at least enable it in forked > repositories (PR builders) which does not consume ASF resource. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48133) Run `sparkr` only in PR builders and Daily CIs
[ https://issues.apache.org/jira/browse/SPARK-48133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48133: -- Fix Version/s: 3.5.2 > Run `sparkr` only in PR builders and Daily CIs > -- > > Key: SPARK-48133 > URL: https://issues.apache.org/jira/browse/SPARK-48133 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48109) Enable `k8s-integration-tests` only for `kubernetes` module change
[ https://issues.apache.org/jira/browse/SPARK-48109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48109: -- Fix Version/s: 3.5.2 > Enable `k8s-integration-tests` only for `kubernetes` module change > -- > > Key: SPARK-48109 > URL: https://issues.apache.org/jira/browse/SPARK-48109 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > Although there is a chance of missing the related core module change, daily > CI test coverage will reveal that. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48116) Run `pyspark-pandas*` only in PR builder and Daily Python CIs
[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48116: -- Fix Version/s: 3.4.4 > Run `pyspark-pandas*` only in PR builder and Daily Python CIs > - > > Key: SPARK-48116 > URL: https://issues.apache.org/jira/browse/SPARK-48116 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48116) Run `pyspark-pandas*` only in PR builder and Daily Python CIs
[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48116: -- Fix Version/s: 3.5.2 > Run `pyspark-pandas*` only in PR builder and Daily Python CIs > - > > Key: SPARK-48116 > URL: https://issues.apache.org/jira/browse/SPARK-48116 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48203) Spin off `pyspark` tests from `build_branch34.yml` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48203. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46480 [https://github.com/apache/spark/pull/46480] > Spin off `pyspark` tests from `build_branch34.yml` Daily CI > --- > > Key: SPARK-48203 > URL: https://issues.apache.org/jira/browse/SPARK-48203 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48202) Spin off `pyspark` tests from `build_branch35.yml` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48202. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46479 [https://github.com/apache/spark/pull/46479] > Spin off `pyspark` tests from `build_branch35.yml` Daily CI > --- > > Key: SPARK-48202 > URL: https://issues.apache.org/jira/browse/SPARK-48202 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-48149. - > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48149. --- Fix Version/s: (was: 4.0.0) Resolution: Abandoned > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-48149: --- > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844713#comment-17844713 ] Dongjoon Hyun commented on SPARK-48149: --- This is technically reverted via SPARK-48200 > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48200) Split `build_python.yml` into per-version cron jobs
[ https://issues.apache.org/jira/browse/SPARK-48200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48200. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46477 [https://github.com/apache/spark/pull/46477] > Split `build_python.yml` into per-version cron jobs > --- > > Key: SPARK-48200 > URL: https://issues.apache.org/jira/browse/SPARK-48200 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48149: - Assignee: (was: Dongjoon Hyun) > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48200) Split `build_python.yml` into per-version cron jobs
[ https://issues.apache.org/jira/browse/SPARK-48200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48200: - Assignee: Dongjoon Hyun > Split `build_python.yml` into per-version cron jobs > --- > > Key: SPARK-48200 > URL: https://issues.apache.org/jira/browse/SPARK-48200 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48200) Split `build_python.yml` into per-version cron jobs
Dongjoon Hyun created SPARK-48200: - Summary: Split `build_python.yml` into per-version cron jobs Key: SPARK-48200 URL: https://issues.apache.org/jira/browse/SPARK-48200 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48198) Upgrade jackson to 2.17.1
[ https://issues.apache.org/jira/browse/SPARK-48198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48198: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade jackson to 2.17.1 > - > > Key: SPARK-48198 > URL: https://issues.apache.org/jira/browse/SPARK-48198 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48198) Upgrade jackson to 2.17.1
[ https://issues.apache.org/jira/browse/SPARK-48198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48198. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46476 [https://github.com/apache/spark/pull/46476] > Upgrade jackson to 2.17.1 > - > > Key: SPARK-48198 > URL: https://issues.apache.org/jira/browse/SPARK-48198 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844690#comment-17844690 ] Dongjoon Hyun commented on SPARK-48094: --- Actually, I'm afraid not because we still have some violations. ASF INFRA policy follows IETF terminology {quote}1. MUST This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification. {quote} And, 3 of 4 policies are `MUST` level like the following. - All workflows MUST have a job concurrency level less than or equal to 20. This means a workflow cannot have more than 20 jobs running at the same time across all matrices. - The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours). - The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours). Let me reopen this. We need to do audit and add a comment to all YAML files in order to prevent a future regression, [~gurwls223] . > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Blocker > Fix For: 4.0.0 > > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-48094: --- Assignee: (was: Dongjoon Hyun) > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Fix For: 4.0.0 > > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48184) Always set the seed of dataframe.sample in Client side
[ https://issues.apache.org/jira/browse/SPARK-48184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48184. --- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull request 46456 [https://github.com/apache/spark/pull/46456] > Always set the seed of dataframe.sample in Client side > -- > > Key: SPARK-48184 > URL: https://issues.apache.org/jira/browse/SPARK-48184 > Project: Spark > Issue Type: Bug > Components: Connect, PySpark >Affects Versions: 4.0.0, 3.5.1, 3.4.3 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 3.5.2, 4.0.0 > > > the output dataframe of `sample` is not immutable in Spark Connect > > In Spark Classic: > {code:java} > In [1]: df = spark.range(1).sample(0.1) > In [2]: [df.count() for i in range(10)] > Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]{code} > > In Spark Connect: > {code:java} > In [1]: df = spark.range(1).sample(0.1) > In [2]: [df.count() for i in range(10)] > Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979] > {code} > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48193) Make `maven-deploy-plugin` retry 3 times
[ https://issues.apache.org/jira/browse/SPARK-48193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48193: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Improvement) > Make `maven-deploy-plugin` retry 3 times > > > Key: SPARK-48193 > URL: https://issues.apache.org/jira/browse/SPARK-48193 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Fix Version/s: 3.4.4 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness, pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48187) Run `docs` only in PR builders and `build_non_ansi` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48187: -- Summary: Run `docs` only in PR builders and `build_non_ansi` Daily CI (was: Run `docs` only in PR builders and Java 21 Daily CI) > Run `docs` only in PR builders and `build_non_ansi` Daily CI > > > Key: SPARK-48187 > URL: https://issues.apache.org/jira/browse/SPARK-48187 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48187) Run `docs` only in PR builders and Java 21 Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48187. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46463 [https://github.com/apache/spark/pull/46463] > Run `docs` only in PR builders and Java 21 Daily CI > --- > > Key: SPARK-48187 > URL: https://issues.apache.org/jira/browse/SPARK-48187 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48187) Run `docs` only in PR builders and Java 21 Daily CI
Dongjoon Hyun created SPARK-48187: - Summary: Run `docs` only in PR builders and Java 21 Daily CI Key: SPARK-48187 URL: https://issues.apache.org/jira/browse/SPARK-48187 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48138) Disable a flaky `SparkSessionE2ESuite.interrupt tag` test
[ https://issues.apache.org/jira/browse/SPARK-48138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48138: -- Fix Version/s: 3.5.2 > Disable a flaky `SparkSessionE2ESuite.interrupt tag` test > - > > Key: SPARK-48138 > URL: https://issues.apache.org/jira/browse/SPARK-48138 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 > (Master, 5/5) > - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 > (Master, 5/4) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48139) Re-enable `SparkSessionE2ESuite.interrupt tag`
[ https://issues.apache.org/jira/browse/SPARK-48139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48139: -- Affects Version/s: 3.5.2 > Re-enable `SparkSessionE2ESuite.interrupt tag` > -- > > Key: SPARK-48139 > URL: https://issues.apache.org/jira/browse/SPARK-48139 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0, 3.5.2 >Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Fix Version/s: 3.5.2 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness, pull-request-available > Fix For: 4.0.0, 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48183) Update error contribution guide to respect new error class file
[ https://issues.apache.org/jira/browse/SPARK-48183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48183. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46455 [https://github.com/apache/spark/pull/46455] > Update error contribution guide to respect new error class file > --- > > Key: SPARK-48183 > URL: https://issues.apache.org/jira/browse/SPARK-48183 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > We moved error class definition from .py to .json but documentation still > shows old behavior. We should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48183) Update error contribution guide to respect new error class file
[ https://issues.apache.org/jira/browse/SPARK-48183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48183: - Assignee: Haejoon Lee > Update error contribution guide to respect new error class file > --- > > Key: SPARK-48183 > URL: https://issues.apache.org/jira/browse/SPARK-48183 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > > We moved error class definition from .py to .json but documentation still > shows old behavior. We should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48152) Make spark-profiler as a part of release and publish to maven central repo
[ https://issues.apache.org/jira/browse/SPARK-48152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48152. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46402 [https://github.com/apache/spark/pull/46402] > Make spark-profiler as a part of release and publish to maven central repo > -- > > Key: SPARK-48152 > URL: https://issues.apache.org/jira/browse/SPARK-48152 > Project: Spark > Issue Type: Improvement > Components: Build, Documentation >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48152) Make spark-profiler as a part of release and publish to maven central repo
[ https://issues.apache.org/jira/browse/SPARK-48152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48152: - Assignee: BingKun Pan > Make spark-profiler as a part of release and publish to maven central repo > -- > > Key: SPARK-48152 > URL: https://issues.apache.org/jira/browse/SPARK-48152 > Project: Spark > Issue Type: Improvement > Components: Build, Documentation >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48178) Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed
[ https://issues.apache.org/jira/browse/SPARK-48178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48178. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46449 [https://github.com/apache/spark/pull/46449] > Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed > -- > > Key: SPARK-48178 > URL: https://issues.apache.org/jira/browse/SPARK-48178 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.5.2 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48178) Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed
[ https://issues.apache.org/jira/browse/SPARK-48178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48178: - Assignee: Dongjoon Hyun > Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed > -- > > Key: SPARK-48178 > URL: https://issues.apache.org/jira/browse/SPARK-48178 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.5.2 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48178) Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed
[ https://issues.apache.org/jira/browse/SPARK-48178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48178: -- Summary: Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed (was: Run `build/scala-211/java-11-17` jobs of branch-3.5 only if needed) > Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed > -- > > Key: SPARK-48178 > URL: https://issues.apache.org/jira/browse/SPARK-48178 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.5.2 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48179) Pin `nbsphinx` to `0.9.3`
[ https://issues.apache.org/jira/browse/SPARK-48179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48179. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46448 [https://github.com/apache/spark/pull/46448] > Pin `nbsphinx` to `0.9.3` > -- > > Key: SPARK-48179 > URL: https://issues.apache.org/jira/browse/SPARK-48179 > Project: Spark > Issue Type: Bug > Components: Project Infra >Affects Versions: 3.5.2 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48179) Pin `nbsphinx` to `0.9.3`
[ https://issues.apache.org/jira/browse/SPARK-48179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48179: - Assignee: Dongjoon Hyun > Pin `nbsphinx` to `0.9.3` > -- > > Key: SPARK-48179 > URL: https://issues.apache.org/jira/browse/SPARK-48179 > Project: Spark > Issue Type: Bug > Components: Project Infra >Affects Versions: 3.5.2 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48179) Pin `nbsphinx` to `0.9.3`
Dongjoon Hyun created SPARK-48179: - Summary: Pin `nbsphinx` to `0.9.3` Key: SPARK-48179 URL: https://issues.apache.org/jira/browse/SPARK-48179 Project: Spark Issue Type: Bug Components: Project Infra Affects Versions: 3.5.2 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48178) Run `build/scala-211/java-11-17` jobs of branch-3.5 only if needed
Dongjoon Hyun created SPARK-48178: - Summary: Run `build/scala-211/java-11-17` jobs of branch-3.5 only if needed Key: SPARK-48178 URL: https://issues.apache.org/jira/browse/SPARK-48178 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 3.5.2 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Upgrade `Parquet` to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Summary: Upgrade `Parquet` to 1.14.0 (was: Bump Parquet to 1.14.0) > Upgrade `Parquet` to 1.14.0 > --- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Affects Version/s: 4.0.0 (was: 3.5.2) > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48177: - Assignee: Fokko Driesprong > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Fix Version/s: (was: 4.0.0) > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48037. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46273 [https://github.com/apache/spark/pull/46273] > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness, pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844388#comment-17844388 ] Dongjoon Hyun commented on SPARK-48037: --- Thank you, [~dzcxzl]. I raised the priority to `Blocker` for all future releases and added a label, `correctness`. > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48037: - Assignee: dzcxzl > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Affects Version/s: 3.4.3 3.5.1 4.0.0 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Target Version/s: 4.0.0, 3.5.2, 3.4.4 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Labels: correctness (was: pull-request-available) > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: dzcxzl >Priority: Major > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Priority: Blocker (was: Major) > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions
[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41547. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46432 [https://github.com/apache/spark/pull/46432] > Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions > -- > > Key: SPARK-41547 > URL: https://issues.apache.org/jira/browse/SPARK-41547 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Xinrong Meng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > See https://issues.apache.org/jira/browse/SPARK-41548 > We should fix the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48169) Use lazy BadRecordException cause for StaxXmlParser and JacksonParser
[ https://issues.apache.org/jira/browse/SPARK-48169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48169. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46438 [https://github.com/apache/spark/pull/46438] > Use lazy BadRecordException cause for StaxXmlParser and JacksonParser > - > > Key: SPARK-48169 > URL: https://issues.apache.org/jira/browse/SPARK-48169 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Vladimir Golubev >Assignee: Vladimir Golubev >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > For now since the https://issues.apache.org/jira/browse/SPARK-48143, the old > constructor is used -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48165) Update `ap-loader` to 3.0-9
[ https://issues.apache.org/jira/browse/SPARK-48165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48165. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46427 [https://github.com/apache/spark/pull/46427] > Update `ap-loader` to 3.0-9 > --- > > Key: SPARK-48165 > URL: https://issues.apache.org/jira/browse/SPARK-48165 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48173) CheckAnalsis should see the entire query plan
[ https://issues.apache.org/jira/browse/SPARK-48173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48173. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46439 [https://github.com/apache/spark/pull/46439] > CheckAnalsis should see the entire query plan > - > > Key: SPARK-48173 > URL: https://issues.apache.org/jira/browse/SPARK-48173 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48173) CheckAnalsis should see the entire query plan
[ https://issues.apache.org/jira/browse/SPARK-48173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48173: - Assignee: Wenchen Fan > CheckAnalsis should see the entire query plan > - > > Key: SPARK-48173 > URL: https://issues.apache.org/jira/browse/SPARK-48173 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48171) Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
[ https://issues.apache.org/jira/browse/SPARK-48171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48171: - Assignee: Yang Jie > Clean up the use of deprecated APIs related to `o.rocksdb.Logger` > - > > Key: SPARK-48171 > URL: https://issues.apache.org/jira/browse/SPARK-48171 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > > {code:java} > /** > * AbstractLogger constructor. > * > * Important: the log level set within > * the {@link org.rocksdb.Options} instance will be used as > * maximum log level of RocksDB. > * > * @param options {@link org.rocksdb.Options} instance. > * > * @deprecated Use {@link Logger#Logger(InfoLogLevel)} instead, e.g. {@code > new > * Logger(options.infoLogLevel())}. > */ > @Deprecated > public Logger(final Options options) { > this(options.infoLogLevel()); > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48171) Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
[ https://issues.apache.org/jira/browse/SPARK-48171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48171. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46436 [https://github.com/apache/spark/pull/46436] > Clean up the use of deprecated APIs related to `o.rocksdb.Logger` > - > > Key: SPARK-48171 > URL: https://issues.apache.org/jira/browse/SPARK-48171 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > {code:java} > /** > * AbstractLogger constructor. > * > * Important: the log level set within > * the {@link org.rocksdb.Options} instance will be used as > * maximum log level of RocksDB. > * > * @param options {@link org.rocksdb.Options} instance. > * > * @deprecated Use {@link Logger#Logger(InfoLogLevel)} instead, e.g. {@code > new > * Logger(options.infoLogLevel())}. > */ > @Deprecated > public Logger(final Options options) { > this(options.infoLogLevel()); > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48163) Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48163. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46425 [https://github.com/apache/spark/pull/46425] > Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > > > Key: SPARK-48163 > URL: https://issues.apache.org/jira/browse/SPARK-48163 > Project: Spark > Issue Type: Sub-task > Components: SQL, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > {code} > - SPARK-43923: commands send events ((get_resources_command { > [info] } > [info] ,None)) *** FAILED *** (35 milliseconds) > [info] VerifyEvents.this.listener.executeHolder.isDefined was false > (SparkConnectServiceSuite.scala:873) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48139) Re-enable `SparkSessionE2ESuite.interrupt tag`
[ https://issues.apache.org/jira/browse/SPARK-48139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48139: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Bug) > Re-enable `SparkSessionE2ESuite.interrupt tag` > -- > > Key: SPARK-48139 > URL: https://issues.apache.org/jira/browse/SPARK-48139 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48164: -- Component/s: Tests > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48164: -- Target Version/s: 4.0.0 > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48163) Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48163: - Assignee: Dongjoon Hyun > Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > > > Key: SPARK-48163 > URL: https://issues.apache.org/jira/browse/SPARK-48163 > Project: Spark > Issue Type: Sub-task > Components: SQL, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > > {code} > - SPARK-43923: commands send events ((get_resources_command { > [info] } > [info] ,None)) *** FAILED *** (35 milliseconds) > [info] VerifyEvents.this.listener.executeHolder.isDefined was false > (SparkConnectServiceSuite.scala:873) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
Dongjoon Hyun created SPARK-48164: - Summary: Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command` Key: SPARK-48164 URL: https://issues.apache.org/jira/browse/SPARK-48164 Project: Spark Issue Type: Sub-task Components: Connect Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48163) Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
Dongjoon Hyun created SPARK-48163: - Summary: Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command` Key: SPARK-48163 URL: https://issues.apache.org/jira/browse/SPARK-48163 Project: Spark Issue Type: Sub-task Components: SQL, Tests Affects Versions: 4.0.0 Reporter: Dongjoon Hyun {code} - SPARK-43923: commands send events ((get_resources_command { [info] } [info] ,None)) *** FAILED *** (35 milliseconds) [info] VerifyEvents.this.listener.executeHolder.isDefined was false (SparkConnectServiceSuite.scala:873) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48141) Update the Oracle docker image version used for test and integration to use Oracle Database 23ai Free
[ https://issues.apache.org/jira/browse/SPARK-48141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48141. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46399 [https://github.com/apache/spark/pull/46399] > Update the Oracle docker image version used for test and integration to use > Oracle Database 23ai Free > - > > Key: SPARK-48141 > URL: https://issues.apache.org/jira/browse/SPARK-48141 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: Luca Canali >Assignee: Luca Canali >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > {color:#0d0d0d}Oracle recently released Oracle Database 23ai Free, > specifically version 23.4, as their latest free database version. {color} > {color:#0d0d0d}We should update our testing infrastructure to utilize this > free version, using the Docker image available at{color}[ {color:#0d0d0d} > {color}|https://github.com/gvenzl/oci-oracle-free] > [https://github.com/gvenzl/oci-oracle-free > |https://github.com/gvenzl/oci-oracle-free] > {color:#0d0d0d}This repository is known for being a reliable and > well-maintained source for Oracle Database images.{color} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48141) Update the Oracle docker image version used for test and integration to use Oracle Database 23ai Free
[ https://issues.apache.org/jira/browse/SPARK-48141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48141: - Assignee: Luca Canali > Update the Oracle docker image version used for test and integration to use > Oracle Database 23ai Free > - > > Key: SPARK-48141 > URL: https://issues.apache.org/jira/browse/SPARK-48141 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: Luca Canali >Assignee: Luca Canali >Priority: Minor > Labels: pull-request-available > > {color:#0d0d0d}Oracle recently released Oracle Database 23ai Free, > specifically version 23.4, as their latest free database version. {color} > {color:#0d0d0d}We should update our testing infrastructure to utilize this > free version, using the Docker image available at{color}[ {color:#0d0d0d} > {color}|https://github.com/gvenzl/oci-oracle-free] > [https://github.com/gvenzl/oci-oracle-free > |https://github.com/gvenzl/oci-oracle-free] > {color:#0d0d0d}This repository is known for being a reliable and > well-maintained source for Oracle Database images.{color} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48150) Fix nullability of try_parse_json
[ https://issues.apache.org/jira/browse/SPARK-48150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48150. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46409 [https://github.com/apache/spark/pull/46409] > Fix nullability of try_parse_json > - > > Key: SPARK-48150 > URL: https://issues.apache.org/jira/browse/SPARK-48150 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Josh Rosen >Assignee: Josh Rosen >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Followup for SPARK-47922: `try_parse_json` must declare a nullable output. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48153) Run `build` job of `build_and_test.yml` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48153. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46412 [https://github.com/apache/spark/pull/46412] > Run `build` job of `build_and_test.yml` only if needed > -- > > Key: SPARK-48153 > URL: https://issues.apache.org/jira/browse/SPARK-48153 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48153) Run `build` job of `build_and_test.yml` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48153: -- Summary: Run `build` job of `build_and_test.yml` only if needed (was: Run `build` job only if needed) > Run `build` job of `build_and_test.yml` only if needed > -- > > Key: SPARK-48153 > URL: https://issues.apache.org/jira/browse/SPARK-48153 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48151) `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5`
[ https://issues.apache.org/jira/browse/SPARK-48151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48151: - Assignee: Dongjoon Hyun > `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5` > > > Key: SPARK-48151 > URL: https://issues.apache.org/jira/browse/SPARK-48151 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48151) `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5`
[ https://issues.apache.org/jira/browse/SPARK-48151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48151. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46410 [https://github.com/apache/spark/pull/46410] > `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5` > > > Key: SPARK-48151 > URL: https://issues.apache.org/jira/browse/SPARK-48151 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48151) `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5`
Dongjoon Hyun created SPARK-48151: - Summary: `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5` Key: SPARK-48151 URL: https://issues.apache.org/jira/browse/SPARK-48151 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48149. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46407 [https://github.com/apache/spark/pull/46407] > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48149: - Assignee: Dongjoon Hyun > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
Dongjoon Hyun created SPARK-48149: - Summary: Serialize `build_python.yml` to run a single Python version per cron schedule Key: SPARK-48149 URL: https://issues.apache.org/jira/browse/SPARK-48149 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48145) Remove logDebug and logTrace with MDC in java structured logging framework
[ https://issues.apache.org/jira/browse/SPARK-48145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48145. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46405 [https://github.com/apache/spark/pull/46405] > Remove logDebug and logTrace with MDC in java structured logging framework > -- > > Key: SPARK-48145 > URL: https://issues.apache.org/jira/browse/SPARK-48145 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48138) Disable a flaky `SparkSessionE2ESuite.interrupt tag` test
[ https://issues.apache.org/jira/browse/SPARK-48138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48138: -- Description: - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 (Master, 5/5) - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 (Master, 5/4) > Disable a flaky `SparkSessionE2ESuite.interrupt tag` test > - > > Key: SPARK-48138 > URL: https://issues.apache.org/jira/browse/SPARK-48138 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > > - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 > (Master, 5/5) > - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 > (Master, 5/4) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48139) Re-enable `SparkSessionE2ESuite.interrupt tag`
[ https://issues.apache.org/jira/browse/SPARK-48139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48139: -- Description: (was: - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 (Master, 5/5) - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 (Master, 5/4)) > Re-enable `SparkSessionE2ESuite.interrupt tag` > -- > > Key: SPARK-48139 > URL: https://issues.apache.org/jira/browse/SPARK-48139 > Project: Spark > Issue Type: Bug > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48139) Re-enable `SparkSessionE2ESuite.interrupt tag`
[ https://issues.apache.org/jira/browse/SPARK-48139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48139: -- Description: - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 (Master, 5/5) - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 (Master, 5/4) > Re-enable `SparkSessionE2ESuite.interrupt tag` > -- > > Key: SPARK-48139 > URL: https://issues.apache.org/jira/browse/SPARK-48139 > Project: Spark > Issue Type: Bug > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > > - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 > (Master, 5/5) > - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 > (Master, 5/4) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48138) Disable a flaky `SparkSessionE2ESuite.interrupt tag` test
Dongjoon Hyun created SPARK-48138: - Summary: Disable a flaky `SparkSessionE2ESuite.interrupt tag` test Key: SPARK-48138 URL: https://issues.apache.org/jira/browse/SPARK-48138 Project: Spark Issue Type: Sub-task Components: Connect, Tests Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48136) Always upload Spark Connect log files
[ https://issues.apache.org/jira/browse/SPARK-48136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48136. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46393 [https://github.com/apache/spark/pull/46393] > Always upload Spark Connect log files > - > > Key: SPARK-48136 > URL: https://issues.apache.org/jira/browse/SPARK-48136 > Project: Spark > Issue Type: Improvement > Components: Connect, Project Infra, PySpark >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > We should always upload log files if it is not success -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48135) Run `buf` and `ui` only in PR builders and Java 21 Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48135: -- Summary: Run `buf` and `ui` only in PR builders and Java 21 Daily CI (was: Run `but` and `ui` only in PR builders and Java 21 Daily CI) > Run `buf` and `ui` only in PR builders and Java 21 Daily CI > --- > > Key: SPARK-48135 > URL: https://issues.apache.org/jira/browse/SPARK-48135 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48132) Run `k8s-integration-tests` only in PR builder and Daily CIs
[ https://issues.apache.org/jira/browse/SPARK-48132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48132: -- Summary: Run `k8s-integration-tests` only in PR builder and Daily CIs (was: Run `k8s-integration-tests` in PR builder and Daily CIs) > Run `k8s-integration-tests` only in PR builder and Daily CIs > > > Key: SPARK-48132 > URL: https://issues.apache.org/jira/browse/SPARK-48132 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48116) Run `pyspark-pandas*` only in PR builder and Daily Python CIs
[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48116: -- Summary: Run `pyspark-pandas*` only in PR builder and Daily Python CIs (was: Run `pyspark-pandas*` in PR builder and Daily Python CIs) > Run `pyspark-pandas*` only in PR builder and Daily Python CIs > - > > Key: SPARK-48116 > URL: https://issues.apache.org/jira/browse/SPARK-48116 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2. MONITORING - https://infra-reports.apache.org/#ghactions=spark=168 !Screenshot 2024-05-02 at 23.56.05.png|width=100%! h2. TARGET * All workflows MUST have a job concurrency level less than or equal to 20. This means a workflow cannot have more than 20 jobs running at the same time across all matrices. * All workflows SHOULD have a job concurrency level less than or equal to 15. Just because 20 is the max, doesn't mean you should strive for 20. * The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours). * The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours). h2. DEADLINE bq. 17th of May, 2024 Since the deadline is 17th of May, 2024, I set this as the highest priority, `Blocker`. was: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2. MONITORING [https://infra-reports.apache.org/#ghactions=spark=168] !Screenshot 2024-05-02 at 23.56.05.png|width=100%! h2. TARGET * All workflows MUST have a job concurrency level less than or equal to 20. This means a workflow cannot have more than 20 jobs running at the same time across all matrices. * All workflows SHOULD have a job concurrency level less than or equal to 15. Just because 20 is the max, doesn't mean you should strive for 20. * The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours). * The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours). h2. DEADLINE bq. 17th of May, 2024 Since the deadline is 17th of May, 2024, I set this as the highest priority, `Blocker`. > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48131) Unify MDC key `mdc.taskName` and `task_name`
[ https://issues.apache.org/jira/browse/SPARK-48131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48131. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46386 [https://github.com/apache/spark/pull/46386] > Unify MDC key `mdc.taskName` and `task_name` > > > Key: SPARK-48131 > URL: https://issues.apache.org/jira/browse/SPARK-48131 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Rename the MDC key `mdc.taskName` as `task_name`, so that it is consistent > with all the MDC keys used in the structured logging framework. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48128) BitwiseCount / bit_count generated code for boolean inputs fails to compile
[ https://issues.apache.org/jira/browse/SPARK-48128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48128: -- Affects Version/s: 3.4.3 3.3.4 3.5.1 3.2.4 3.1.3 > BitwiseCount / bit_count generated code for boolean inputs fails to compile > --- > > Key: SPARK-48128 > URL: https://issues.apache.org/jira/browse/SPARK-48128 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 3.1.3, 3.2.4, 3.5.1, 3.3.4, 3.4.3 >Reporter: Josh Rosen >Assignee: Josh Rosen >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > > If the `BitwiseCount` / `bit_count` expresison is applied to a boolean type > column then then it will trigger codegen fallback to interpreted because the > generated code contains invalid Java syntax, triggering errors like > {code} > java.util.concurrent.ExecutionException: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 41, Column 11: Failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 41, Column 11: Unexpected token "if" in primary > {code} > This problem was masked because the QueryTest framework may not be fully > exercising codegen paths (e.g. if constant folding occurs). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48128) BitwiseCount / bit_count generated code for boolean inputs fails to compile
[ https://issues.apache.org/jira/browse/SPARK-48128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48128: -- Issue Type: Bug (was: Improvement) > BitwiseCount / bit_count generated code for boolean inputs fails to compile > --- > > Key: SPARK-48128 > URL: https://issues.apache.org/jira/browse/SPARK-48128 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0 >Reporter: Josh Rosen >Assignee: Josh Rosen >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > > If the `BitwiseCount` / `bit_count` expresison is applied to a boolean type > column then then it will trigger codegen fallback to interpreted because the > generated code contains invalid Java syntax, triggering errors like > {code} > java.util.concurrent.ExecutionException: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 41, Column 11: Failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 41, Column 11: Unexpected token "if" in primary > {code} > This problem was masked because the QueryTest framework may not be fully > exercising codegen paths (e.g. if constant folding occurs). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48129) Provide a constant table schema in PySpark for querying structured logs
[ https://issues.apache.org/jira/browse/SPARK-48129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48129. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46384 [https://github.com/apache/spark/pull/46384] > Provide a constant table schema in PySpark for querying structured logs > --- > > Key: SPARK-48129 > URL: https://issues.apache.org/jira/browse/SPARK-48129 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48128) BitwiseCount / bit_count generated code for boolean inputs fails to compile
[ https://issues.apache.org/jira/browse/SPARK-48128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48128. --- Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull request 46382 [https://github.com/apache/spark/pull/46382] > BitwiseCount / bit_count generated code for boolean inputs fails to compile > --- > > Key: SPARK-48128 > URL: https://issues.apache.org/jira/browse/SPARK-48128 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Josh Rosen >Assignee: Josh Rosen >Priority: Major > Labels: pull-request-available > Fix For: 3.4.4, 3.5.2, 4.0.0 > > > If the `BitwiseCount` / `bit_count` expresison is applied to a boolean type > column then then it will trigger codegen fallback to interpreted because the > generated code contains invalid Java syntax, triggering errors like > {code} > java.util.concurrent.ExecutionException: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 41, Column 11: Failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 41, Column 11: Unexpected token "if" in primary > {code} > This problem was masked because the QueryTest framework may not be fully > exercising codegen paths (e.g. if constant folding occurs). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48124) Disable structured logging for Interpreter by default
[ https://issues.apache.org/jira/browse/SPARK-48124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48124. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46383 [https://github.com/apache/spark/pull/46383] > Disable structured logging for Interpreter by default > - > > Key: SPARK-48124 > URL: https://issues.apache.org/jira/browse/SPARK-48124 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 4.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Since there are plain text output from > Interpreters(spark-shell/spark-sql/pyspark), it makes more sense to disable > structured logging for Interpreters by default. > > spark-shell output when with structured logging enabled: > ``` > Setting default log level to "WARN". > To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use > setLogLevel(newLevel). > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT > /_/ > > Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9) > Type in expressions to have them evaluated. > Type :help for more information. > {"ts":"2024-05-04T01:11:03.797Z","level":"WARN","msg":"Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable","logger":"NativeCodeLoader"} > {"ts":"2024-05-04T01:11:04.104Z","level":"WARN","msg":"Service 'SparkUI' > could not bind on port 4040. Attempting port 4041.","logger":"Utils"} > Spark context Web UI available at http://10.10.114.155:4041 > Spark context available as 'sc' (master = local[*], app id = > local-1714785064155). > Spark session available as 'spark'. > ``` > > spark-shell output when without structured logging enabled: > ``` > Setting default log level to "WARN". > To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use > setLogLevel(newLevel). > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT > /_/ > > Using Scala version 2.13.13 (OpenJDK 64-Bit Server VM, Java 17.0.9) > Type in expressions to have them evaluated. > Type :help for more information. > 24/05/03 18:11:35 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 24/05/03 18:11:35 WARN Utils: Service 'SparkUI' could not bind on port 4040. > Attempting port 4041. > Spark context Web UI available at http://10.10.114.155:4041 > Spark context available as 'sc' (master = local[*], app id = > local-1714785095892). > Spark session available as 'spark'. > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48127) Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules
[ https://issues.apache.org/jira/browse/SPARK-48127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48127. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46376 [https://github.com/apache/spark/pull/46376] > Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules > --- > > Key: SPARK-48127 > URL: https://issues.apache.org/jira/browse/SPARK-48127 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org