[jira] [Resolved] (SPARK-48227) Document the requirement of seed in protos
[ https://issues.apache.org/jira/browse/SPARK-48227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48227. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46518 [https://github.com/apache/spark/pull/46518] > Document the requirement of seed in protos > -- > > Key: SPARK-48227 > URL: https://issues.apache.org/jira/browse/SPARK-48227 > Project: Spark > Issue Type: Improvement > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48227) Document the requirement of seed in protos
[ https://issues.apache.org/jira/browse/SPARK-48227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48227: - Assignee: Ruifeng Zheng > Document the requirement of seed in protos > -- > > Key: SPARK-48227 > URL: https://issues.apache.org/jira/browse/SPARK-48227 > Project: Spark > Issue Type: Improvement > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48226) Add `spark-ganglia-lgpl` to `lint-java` & `spark-ganglia-lgpl` and `jvm-profiler` to `sbt-checkstyle`
[ https://issues.apache.org/jira/browse/SPARK-48226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48226. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46501 [https://github.com/apache/spark/pull/46501] > Add `spark-ganglia-lgpl` to `lint-java` & `spark-ganglia-lgpl` and > `jvm-profiler` to `sbt-checkstyle` > - > > Key: SPARK-48226 > URL: https://issues.apache.org/jira/browse/SPARK-48226 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48226) Add `spark-ganglia-lgpl` to `lint-java` & `spark-ganglia-lgpl` and `jvm-profiler` to `sbt-checkstyle`
[ https://issues.apache.org/jira/browse/SPARK-48226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48226: - Assignee: BingKun Pan > Add `spark-ganglia-lgpl` to `lint-java` & `spark-ganglia-lgpl` and > `jvm-profiler` to `sbt-checkstyle` > - > > Key: SPARK-48226 > URL: https://issues.apache.org/jira/browse/SPARK-48226 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48225) Upgrade `sbt` to 1.10.0
Dongjoon Hyun created SPARK-48225: - Summary: Upgrade `sbt` to 1.10.0 Key: SPARK-48225 URL: https://issues.apache.org/jira/browse/SPARK-48225 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-48164. - > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Bug > Components: Connect, Tests >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48164: -- Parent: (was: SPARK-44111) Issue Type: Bug (was: Sub-task) > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Bug > Components: Connect, Tests >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48164: -- Priority: Major (was: Blocker) > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47930) Upgrade RoaringBitmap to 1.0.6
[ https://issues.apache.org/jira/browse/SPARK-47930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47930: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade RoaringBitmap to 1.0.6 > -- > > Key: SPARK-47930 > URL: https://issues.apache.org/jira/browse/SPARK-47930 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47982) Update code style' plugins to latest version
[ https://issues.apache.org/jira/browse/SPARK-47982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47982: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Update code style' plugins to latest version > > > Key: SPARK-47982 > URL: https://issues.apache.org/jira/browse/SPARK-47982 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
Re: [DISCUSS] Spark 4.0.0 release
Please re-try to upload, Wenchen. ASF Infra team bumped up our upload limit based on our request. > Your upload limit has been increased to 650MB Dongjoon. On Thu, May 9, 2024 at 8:12 AM Wenchen Fan wrote: > I've created a ticket: https://issues.apache.org/jira/browse/INFRA-25776 > > On Thu, May 9, 2024 at 11:06 PM Dongjoon Hyun > wrote: > >> In addition, FYI, I was the latest release manager with Apache Spark >> 3.4.3 (2024-04-15 Vote) >> >> According to my work log, I uploaded the following binaries to SVN from >> EC2 (us-west-2) without any issues. >> >> -rw-r--r--. 1 centos centos 311384003 Apr 15 01:29 pyspark-3.4.3.tar.gz >> -rw-r--r--. 1 centos centos 397870995 Apr 15 00:44 >> spark-3.4.3-bin-hadoop3-scala2.13.tgz >> -rw-r--r--. 1 centos centos 388930980 Apr 15 01:29 >> spark-3.4.3-bin-hadoop3.tgz >> -rw-r--r--. 1 centos centos 300786123 Apr 15 01:04 >> spark-3.4.3-bin-without-hadoop.tgz >> -rw-r--r--. 1 centos centos 32219044 Apr 15 00:23 spark-3.4.3.tgz >> -rw-r--r--. 1 centos centos356749 Apr 15 01:29 SparkR_3.4.3.tar.gz >> >> Since Apache Spark 4.0.0-preview doesn't have Scala 2.12 combination, the >> total size should be smaller than 3.4.3 binaires. >> >> Given that, if there is any INFRA change, that could happen after 4/15. >> >> Dongjoon. >> >> On Thu, May 9, 2024 at 7:57 AM Dongjoon Hyun >> wrote: >> >>> Could you file an INFRA JIRA issue with the error message and context >>> first, Wenchen? >>> >>> As you know, if we see something, we had better file a JIRA issue >>> because it could be not only an Apache Spark project issue but also all ASF >>> project issues. >>> >>> Dongjoon. >>> >>> >>> On Thu, May 9, 2024 at 12:28 AM Wenchen Fan wrote: >>> >>>> UPDATE: >>>> >>>> After resolving a few issues in the release scripts, I can finally >>>> build the release packages. However, I can't upload them to the staging SVN >>>> repo due to a transmitting error, and it seems like a limitation from the >>>> server side. I tried it on both my local laptop and remote AWS instance, >>>> but neither works. These package binaries are like 300-400 MBs, and we just >>>> did a release last month. Not sure if this is a new limitation due to cost >>>> saving. >>>> >>>> While I'm looking for help to get unblocked, I'm wondering if we can >>>> upload release packages to a public git repo instead, under the Apache >>>> account? >>>> >>>>> >>>>>>>>>>>>>
[jira] [Resolved] (SPARK-48216) Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related tests configurable
[ https://issues.apache.org/jira/browse/SPARK-48216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48216. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46505 [https://github.com/apache/spark/pull/46505] > Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related > tests configurable > > > Key: SPARK-48216 > URL: https://issues.apache.org/jira/browse/SPARK-48216 > Project: Spark > Issue Type: Sub-task > Components: Spark Docker, Tests >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48216) Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related tests configurable
[ https://issues.apache.org/jira/browse/SPARK-48216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48216: - Assignee: Kent Yao > Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related > tests configurable > > > Key: SPARK-48216 > URL: https://issues.apache.org/jira/browse/SPARK-48216 > Project: Spark > Issue Type: Sub-task > Components: Spark Docker, Tests >Affects Versions: 4.0.0 >Reporter: Kent Yao >Assignee: Kent Yao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
Re: [DISCUSS] Spark 4.0.0 release
In addition, FYI, I was the latest release manager with Apache Spark 3.4.3 (2024-04-15 Vote) According to my work log, I uploaded the following binaries to SVN from EC2 (us-west-2) without any issues. -rw-r--r--. 1 centos centos 311384003 Apr 15 01:29 pyspark-3.4.3.tar.gz -rw-r--r--. 1 centos centos 397870995 Apr 15 00:44 spark-3.4.3-bin-hadoop3-scala2.13.tgz -rw-r--r--. 1 centos centos 388930980 Apr 15 01:29 spark-3.4.3-bin-hadoop3.tgz -rw-r--r--. 1 centos centos 300786123 Apr 15 01:04 spark-3.4.3-bin-without-hadoop.tgz -rw-r--r--. 1 centos centos 32219044 Apr 15 00:23 spark-3.4.3.tgz -rw-r--r--. 1 centos centos356749 Apr 15 01:29 SparkR_3.4.3.tar.gz Since Apache Spark 4.0.0-preview doesn't have Scala 2.12 combination, the total size should be smaller than 3.4.3 binaires. Given that, if there is any INFRA change, that could happen after 4/15. Dongjoon. On Thu, May 9, 2024 at 7:57 AM Dongjoon Hyun wrote: > Could you file an INFRA JIRA issue with the error message and context > first, Wenchen? > > As you know, if we see something, we had better file a JIRA issue because > it could be not only an Apache Spark project issue but also all ASF project > issues. > > Dongjoon. > > > On Thu, May 9, 2024 at 12:28 AM Wenchen Fan wrote: > >> UPDATE: >> >> After resolving a few issues in the release scripts, I can finally build >> the release packages. However, I can't upload them to the staging SVN repo >> due to a transmitting error, and it seems like a limitation from the server >> side. I tried it on both my local laptop and remote AWS instance, but >> neither works. These package binaries are like 300-400 MBs, and we just did >> a release last month. Not sure if this is a new limitation due to cost >> saving. >> >> While I'm looking for help to get unblocked, I'm wondering if we can >> upload release packages to a public git repo instead, under the Apache >> account? >> >>> >>>>>>>>>>>
Re: [DISCUSS] Spark 4.0.0 release
;>>> safe >>>>>>> (there was some concern from earlier release processes). >>>>>>> >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> >>>>>>> >>>>>>> On Tue, May 7, 2024 at 10:55 AM Nimrod Ofek >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Sorry for the novice question, Wenchen - the release is done >>>>>>>> manually from a laptop? Not using a CI CD process on a build server? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Nimrod >>>>>>>> >>>>>>>> On Tue, May 7, 2024 at 8:50 PM Wenchen Fan >>>>>>>> wrote: >>>>>>>> >>>>>>>>> UPDATE: >>>>>>>>> >>>>>>>>> Unfortunately, it took me quite some time to set up my laptop and >>>>>>>>> get it ready for the release process (docker desktop doesn't work >>>>>>>>> anymore, >>>>>>>>> my pgp key is lost, etc.). I'll start the RC process at my tomorrow. >>>>>>>>> Thanks >>>>>>>>> for your patience! >>>>>>>>> >>>>>>>>> Wenchen >>>>>>>>> >>>>>>>>> On Fri, May 3, 2024 at 7:47 AM yangjie01 >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *发件人**: *Jungtaek Lim >>>>>>>>>> *日期**: *2024年5月2日 星期四 10:21 >>>>>>>>>> *收件人**: *Holden Karau >>>>>>>>>> *抄送**: *Chao Sun , Xiao Li < >>>>>>>>>> gatorsm...@gmail.com>, Tathagata Das , >>>>>>>>>> Wenchen Fan , Cheng Pan , >>>>>>>>>> Nicholas Chammas , Dongjoon Hyun < >>>>>>>>>> dongjoon.h...@gmail.com>, Cheng Pan , Spark >>>>>>>>>> dev list , Anish Shrigondekar < >>>>>>>>>> anish.shrigonde...@databricks.com> >>>>>>>>>> *主题**: *Re: [DISCUSS] Spark 4.0.0 release >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> +1 love to see it! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 2, 2024 at 10:08 AM Holden Karau < >>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> +1 :) yay previews >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 1, 2024 at 5:36 PM Chao Sun >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 1, 2024 at 5:23 PM Xiao Li >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> +1 for next Monday. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We can do more previews when the other features are ready for >>>>>>>>>> preview. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Tathagata Das 于2024年5月1日周三 08:46写道： >>>>>>>>>> >>>>>>>>>> Next week sounds great! Thank you Wenchen! >>>>>>>>>> >>>>>>>>>> >>>>&g
[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[
https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-48094:
--
Description:
h2. ASF INFRA POLICY
- [https://infra.apache.org/github-actions-policy.html]
h2. MONITORING
- [https://infra-reports.apache.org/#ghactions=spark=168]
!Screenshot 2024-05-02 at 23.56.05.png|width=100!
h2. TARGET
* All workflows MUST have a job concurrency level less than or equal to 20.
This means a workflow cannot have more than 20 jobs running at the same time
across all matrices.
* All workflows SHOULD have a job concurrency level less than or equal to 15.
Just because 20 is the max, doesn't mean you should strive for 20.
* The average number of minutes a project uses per calendar week MUST NOT
exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).
* The average number of minutes a project uses in any consecutive five-day
period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes,
or 3,600 hours).
h2. DEADLINE
{quote}17th of May, 2024
{quote}
was:
h2. ASF INFRA POLICY
- https://infra.apache.org/github-actions-policy.html
h2. MONITORING
- https://infra-reports.apache.org/#ghactions=spark=168
!Screenshot 2024-05-02 at 23.56.05.png|width=100%!
h2. TARGET
* All workflows MUST have a job concurrency level less than or equal to 20.
This means a workflow cannot have more than 20 jobs running at the same time
across all matrices.
* All workflows SHOULD have a job concurrency level less than or equal to 15.
Just because 20 is the max, doesn't mean you should strive for 20.
* The average number of minutes a project uses per calendar week MUST NOT
exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).
* The average number of minutes a project uses in any consecutive five-day
period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes,
or 3,600 hours).
h2. DEADLINE
bq. 17th of May, 2024
Since the deadline is 17th of May, 2024, I set this as the highest priority,
`Blocker`.
> Reduce GitHub Action usage according to ASF project allowance
> -
>
> Key: SPARK-48094
> URL: https://issues.apache.org/jira/browse/SPARK-48094
> Project: Spark
> Issue Type: Umbrella
> Components: Project Infra
>Affects Versions: 4.0.0
> Reporter: Dongjoon Hyun
>Priority: Major
> Attachments: Screenshot 2024-05-02 at 23.56.05.png
>
>
> h2. ASF INFRA POLICY
> - [https://infra.apache.org/github-actions-policy.html]
> h2. MONITORING
> - [https://infra-reports.apache.org/#ghactions=spark=168]
> !Screenshot 2024-05-02 at 23.56.05.png|width=100!
> h2. TARGET
> * All workflows MUST have a job concurrency level less than or equal to 20.
> This means a workflow cannot have more than 20 jobs running at the same time
> across all matrices.
> * All workflows SHOULD have a job concurrency level less than or equal to
> 15. Just because 20 is the max, doesn't mean you should strive for 20.
> * The average number of minutes a project uses per calendar week MUST NOT
> exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200
> hours).
> * The average number of minutes a project uses in any consecutive five-day
> period MUST NOT exceed the equivalent of 30 full-time runners (216,000
> minutes, or 3,600 hours).
> h2. DEADLINE
> {quote}17th of May, 2024
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Priority: Major (was: Blocker) > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Fix Version/s: (was: 4.0.0) > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Blocker > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48187) Run `docs` only in PR builders and `build_non_ansi` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48187: -- Fix Version/s: 4.0.0 > Run `docs` only in PR builders and `build_non_ansi` Daily CI > > > Key: SPARK-48187 > URL: https://issues.apache.org/jira/browse/SPARK-48187 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48187) Run `docs` only in PR builders and `build_non_ansi` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48187: -- Fix Version/s: (was: 4.0.0) > Run `docs` only in PR builders and `build_non_ansi` Daily CI > > > Key: SPARK-48187 > URL: https://issues.apache.org/jira/browse/SPARK-48187 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-48094: --- Assignee: (was: Dongjoon Hyun) > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Blocker > Fix For: 4.0.0 > > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48204) fix release script for Spark 4.0+
[ https://issues.apache.org/jira/browse/SPARK-48204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48204: - Assignee: Wenchen Fan > fix release script for Spark 4.0+ > - > > Key: SPARK-48204 > URL: https://issues.apache.org/jira/browse/SPARK-48204 > Project: Spark > Issue Type: Bug > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48204) fix release script for Spark 4.0+
[ https://issues.apache.org/jira/browse/SPARK-48204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48204. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46484 [https://github.com/apache/spark/pull/46484] > fix release script for Spark 4.0+ > - > > Key: SPARK-48204 > URL: https://issues.apache.org/jira/browse/SPARK-48204 > Project: Spark > Issue Type: Bug > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48094. --- Assignee: Dongjoon Hyun Resolution: Fixed > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Blocker > Fix For: 4.0.0 > > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48207) Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48207. --- Fix Version/s: 3.4.4 Resolution: Fixed Issue resolved by pull request 46489 [https://github.com/apache/spark/pull/46489] > Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed > > > Key: SPARK-48207 > URL: https://issues.apache.org/jira/browse/SPARK-48207 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.4 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48207) Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48207: - Assignee: Dongjoon Hyun > Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed > > > Key: SPARK-48207 > URL: https://issues.apache.org/jira/browse/SPARK-48207 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.4 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48207) Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed
[ https://issues.apache.org/jira/browse/SPARK-48207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48207: -- Summary: Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed (was: Run build/scala-213/java-11-17 jobs of `branch-3.4` only if needed) > Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed > > > Key: SPARK-48207 > URL: https://issues.apache.org/jira/browse/SPARK-48207 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.4.4 > Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48207) Run build/scala-213/java-11-17 jobs of `branch-3.4` only if needed
Dongjoon Hyun created SPARK-48207: - Summary: Run build/scala-213/java-11-17 jobs of `branch-3.4` only if needed Key: SPARK-48207 URL: https://issues.apache.org/jira/browse/SPARK-48207 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 3.4.4 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48192) Enable TPC-DS and docker tests in forked repository
[ https://issues.apache.org/jira/browse/SPARK-48192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48192: -- Fix Version/s: 3.4.4 > Enable TPC-DS and docker tests in forked repository > --- > > Key: SPARK-48192 > URL: https://issues.apache.org/jira/browse/SPARK-48192 > Project: Spark > Issue Type: Sub-task > Components: Project Infra, SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > > TPC-DS is pretty important in SQL. Shoud at least enable it in forked > repositories (PR builders) which does not consume ASF resource. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48132) Run `k8s-integration-tests` only in PR builder and Daily CIs
[ https://issues.apache.org/jira/browse/SPARK-48132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48132: -- Fix Version/s: 3.4.4 > Run `k8s-integration-tests` only in PR builder and Daily CIs > > > Key: SPARK-48132 > URL: https://issues.apache.org/jira/browse/SPARK-48132 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48192) Enable TPC-DS and docker tests in forked repository
[ https://issues.apache.org/jira/browse/SPARK-48192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48192: -- Summary: Enable TPC-DS and docker tests in forked repository (was: Enable TPC-DS tests in forked repository) > Enable TPC-DS and docker tests in forked repository > --- > > Key: SPARK-48192 > URL: https://issues.apache.org/jira/browse/SPARK-48192 > Project: Spark > Issue Type: Sub-task > Components: Project Infra, SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > TPC-DS is pretty important in SQL. Shoud at least enable it in forked > repositories (PR builders) which does not consume ASF resource. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48192) Enable TPC-DS tests in forked repository
[ https://issues.apache.org/jira/browse/SPARK-48192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48192: -- Fix Version/s: 3.5.2 > Enable TPC-DS tests in forked repository > > > Key: SPARK-48192 > URL: https://issues.apache.org/jira/browse/SPARK-48192 > Project: Spark > Issue Type: Sub-task > Components: Project Infra, SQL >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > TPC-DS is pretty important in SQL. Shoud at least enable it in forked > repositories (PR builders) which does not consume ASF resource. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48133) Run `sparkr` only in PR builders and Daily CIs
[ https://issues.apache.org/jira/browse/SPARK-48133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48133: -- Fix Version/s: 3.5.2 > Run `sparkr` only in PR builders and Daily CIs > -- > > Key: SPARK-48133 > URL: https://issues.apache.org/jira/browse/SPARK-48133 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48109) Enable `k8s-integration-tests` only for `kubernetes` module change
[ https://issues.apache.org/jira/browse/SPARK-48109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48109: -- Fix Version/s: 3.5.2 > Enable `k8s-integration-tests` only for `kubernetes` module change > -- > > Key: SPARK-48109 > URL: https://issues.apache.org/jira/browse/SPARK-48109 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > Although there is a chance of missing the related core module change, daily > CI test coverage will reveal that. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
Re: Regarding ORC release 2.0.1
Thank you, William. It sounds good to me. I only cherry-picked ORC-1714 from main to branch-2.0 for Apache ORC 2.0.1 release. Dongjoon. On 2024/05/08 06:59:38 "William H." wrote: > Hello All, > > I am preparing to release ORC version 2.0.1 this Thursday, May 9th. > Please let me know if there is anything that you would like to see > included in this release. > > Best Regards, > William >
[jira] [Updated] (SPARK-48116) Run `pyspark-pandas*` only in PR builder and Daily Python CIs
[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48116: -- Fix Version/s: 3.4.4 > Run `pyspark-pandas*` only in PR builder and Daily Python CIs > - > > Key: SPARK-48116 > URL: https://issues.apache.org/jira/browse/SPARK-48116 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48116) Run `pyspark-pandas*` only in PR builder and Daily Python CIs
[ https://issues.apache.org/jira/browse/SPARK-48116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48116: -- Fix Version/s: 3.5.2 > Run `pyspark-pandas*` only in PR builder and Daily Python CIs > - > > Key: SPARK-48116 > URL: https://issues.apache.org/jira/browse/SPARK-48116 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48203) Spin off `pyspark` tests from `build_branch34.yml` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48203. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46480 [https://github.com/apache/spark/pull/46480] > Spin off `pyspark` tests from `build_branch34.yml` Daily CI > --- > > Key: SPARK-48203 > URL: https://issues.apache.org/jira/browse/SPARK-48203 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48202) Spin off `pyspark` tests from `build_branch35.yml` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48202. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46479 [https://github.com/apache/spark/pull/46479] > Spin off `pyspark` tests from `build_branch35.yml` Daily CI > --- > > Key: SPARK-48202 > URL: https://issues.apache.org/jira/browse/SPARK-48202 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (ORC-1715) Bump org.objenesis:objenesis to 3.3
[ https://issues.apache.org/jira/browse/ORC-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1715. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull request 1927 [https://github.com/apache/orc/pull/1927] > Bump org.objenesis:objenesis to 3.3 > --- > > Key: ORC-1715 > URL: https://issues.apache.org/jira/browse/ORC-1715 > Project: ORC > Issue Type: Bug > Components: Java >Affects Versions: 2.0.1 >Reporter: William Hyun >Assignee: William Hyun >Priority: Minor > Fix For: 2.0.1, 2.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-48149. - > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48149. --- Fix Version/s: (was: 4.0.0) Resolution: Abandoned > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-48149: --- > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844713#comment-17844713 ] Dongjoon Hyun commented on SPARK-48149: --- This is technically reverted via SPARK-48200 > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48200) Split `build_python.yml` into per-version cron jobs
[ https://issues.apache.org/jira/browse/SPARK-48200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48200. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46477 [https://github.com/apache/spark/pull/46477] > Split `build_python.yml` into per-version cron jobs > --- > > Key: SPARK-48200 > URL: https://issues.apache.org/jira/browse/SPARK-48200 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48149) Serialize `build_python.yml` to run a single Python version per cron schedule
[ https://issues.apache.org/jira/browse/SPARK-48149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48149: - Assignee: (was: Dongjoon Hyun) > Serialize `build_python.yml` to run a single Python version per cron schedule > - > > Key: SPARK-48149 > URL: https://issues.apache.org/jira/browse/SPARK-48149 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48200) Split `build_python.yml` into per-version cron jobs
[ https://issues.apache.org/jira/browse/SPARK-48200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48200: - Assignee: Dongjoon Hyun > Split `build_python.yml` into per-version cron jobs > --- > > Key: SPARK-48200 > URL: https://issues.apache.org/jira/browse/SPARK-48200 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48200) Split `build_python.yml` into per-version cron jobs
Dongjoon Hyun created SPARK-48200: - Summary: Split `build_python.yml` into per-version cron jobs Key: SPARK-48200 URL: https://issues.apache.org/jira/browse/SPARK-48200 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48198) Upgrade jackson to 2.17.1
[ https://issues.apache.org/jira/browse/SPARK-48198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48198: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade jackson to 2.17.1 > - > > Key: SPARK-48198 > URL: https://issues.apache.org/jira/browse/SPARK-48198 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48198) Upgrade jackson to 2.17.1
[ https://issues.apache.org/jira/browse/SPARK-48198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48198. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46476 [https://github.com/apache/spark/pull/46476] > Upgrade jackson to 2.17.1 > - > > Key: SPARK-48198 > URL: https://issues.apache.org/jira/browse/SPARK-48198 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[
https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844690#comment-17844690
]
Dongjoon Hyun commented on SPARK-48094:
---
Actually, I'm afraid not because we still have some violations.
ASF INFRA policy follows IETF terminology
{quote}1. MUST This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification.
{quote}
And, 3 of 4 policies are `MUST` level like the following.
- All workflows MUST have a job concurrency level less than or equal to 20.
This means a workflow cannot have more than 20 jobs running at the same time
across all matrices.
- The average number of minutes a project uses per calendar week MUST NOT
exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).
- The average number of minutes a project uses in any consecutive five-day
period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes,
or 3,600 hours).
Let me reopen this. We need to do audit and add a comment to all YAML files in
order to prevent a future regression, [~gurwls223] .
> Reduce GitHub Action usage according to ASF project allowance
> -
>
> Key: SPARK-48094
> URL: https://issues.apache.org/jira/browse/SPARK-48094
> Project: Spark
> Issue Type: Umbrella
> Components: Project Infra
>Affects Versions: 4.0.0
> Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: Screenshot 2024-05-02 at 23.56.05.png
>
>
> h2. ASF INFRA POLICY
> - https://infra.apache.org/github-actions-policy.html
> h2. MONITORING
> - https://infra-reports.apache.org/#ghactions=spark=168
> !Screenshot 2024-05-02 at 23.56.05.png|width=100%!
> h2. TARGET
> * All workflows MUST have a job concurrency level less than or equal to 20.
> This means a workflow cannot have more than 20 jobs running at the same time
> across all matrices.
> * All workflows SHOULD have a job concurrency level less than or equal to 15.
> Just because 20 is the max, doesn't mean you should strive for 20.
> * The average number of minutes a project uses per calendar week MUST NOT
> exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200
> hours).
> * The average number of minutes a project uses in any consecutive five-day
> period MUST NOT exceed the equivalent of 30 full-time runners (216,000
> minutes, or 3,600 hours).
> h2. DEADLINE
> bq. 17th of May, 2024
> Since the deadline is 17th of May, 2024, I set this as the highest priority,
> `Blocker`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance
[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-48094: --- Assignee: (was: Dongjoon Hyun) > Reduce GitHub Action usage according to ASF project allowance > - > > Key: SPARK-48094 > URL: https://issues.apache.org/jira/browse/SPARK-48094 > Project: Spark > Issue Type: Umbrella > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Blocker > Fix For: 4.0.0 > > Attachments: Screenshot 2024-05-02 at 23.56.05.png > > > h2. ASF INFRA POLICY > - https://infra.apache.org/github-actions-policy.html > h2. MONITORING > - https://infra-reports.apache.org/#ghactions=spark=168 > !Screenshot 2024-05-02 at 23.56.05.png|width=100%! > h2. TARGET > * All workflows MUST have a job concurrency level less than or equal to 20. > This means a workflow cannot have more than 20 jobs running at the same time > across all matrices. > * All workflows SHOULD have a job concurrency level less than or equal to 15. > Just because 20 is the max, doesn't mean you should strive for 20. > * The average number of minutes a project uses per calendar week MUST NOT > exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 > hours). > * The average number of minutes a project uses in any consecutive five-day > period MUST NOT exceed the equivalent of 30 full-time runners (216,000 > minutes, or 3,600 hours). > h2. DEADLINE > bq. 17th of May, 2024 > Since the deadline is 17th of May, 2024, I set this as the highest priority, > `Blocker`. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48184) Always set the seed of dataframe.sample in Client side
[
https://issues.apache.org/jira/browse/SPARK-48184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-48184.
---
Fix Version/s: 3.5.2
4.0.0
Resolution: Fixed
Issue resolved by pull request 46456
[https://github.com/apache/spark/pull/46456]
> Always set the seed of dataframe.sample in Client side
> --
>
> Key: SPARK-48184
> URL: https://issues.apache.org/jira/browse/SPARK-48184
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
>Affects Versions: 4.0.0, 3.5.1, 3.4.3
>Reporter: Ruifeng Zheng
>Assignee: Ruifeng Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.2, 4.0.0
>
>
> the output dataframe of `sample` is not immutable in Spark Connect
>
> In Spark Classic:
> {code:java}
> In [1]: df = spark.range(1).sample(0.1)
> In [2]: [df.count() for i in range(10)]
> Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]{code}
>
> In Spark Connect:
> {code:java}
> In [1]: df = spark.range(1).sample(0.1)
> In [2]: [df.count() for i in range(10)]
> Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979]
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48193) Make `maven-deploy-plugin` retry 3 times
[ https://issues.apache.org/jira/browse/SPARK-48193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48193: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Improvement) > Make `maven-deploy-plugin` retry 3 times > > > Key: SPARK-48193 > URL: https://issues.apache.org/jira/browse/SPARK-48193 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Fix Version/s: 3.4.4 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness, pull-request-available > Fix For: 4.0.0, 3.5.2, 3.4.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48187) Run `docs` only in PR builders and `build_non_ansi` Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48187: -- Summary: Run `docs` only in PR builders and `build_non_ansi` Daily CI (was: Run `docs` only in PR builders and Java 21 Daily CI) > Run `docs` only in PR builders and `build_non_ansi` Daily CI > > > Key: SPARK-48187 > URL: https://issues.apache.org/jira/browse/SPARK-48187 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48187) Run `docs` only in PR builders and Java 21 Daily CI
[ https://issues.apache.org/jira/browse/SPARK-48187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48187. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46463 [https://github.com/apache/spark/pull/46463] > Run `docs` only in PR builders and Java 21 Daily CI > --- > > Key: SPARK-48187 > URL: https://issues.apache.org/jira/browse/SPARK-48187 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48187) Run `docs` only in PR builders and Java 21 Daily CI
Dongjoon Hyun created SPARK-48187: - Summary: Run `docs` only in PR builders and Java 21 Daily CI Key: SPARK-48187 URL: https://issues.apache.org/jira/browse/SPARK-48187 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48138) Disable a flaky `SparkSessionE2ESuite.interrupt tag` test
[ https://issues.apache.org/jira/browse/SPARK-48138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48138: -- Fix Version/s: 3.5.2 > Disable a flaky `SparkSessionE2ESuite.interrupt tag` test > - > > Key: SPARK-48138 > URL: https://issues.apache.org/jira/browse/SPARK-48138 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.2 > > > - https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 > (Master, 5/5) > - https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 > (Master, 5/4) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48139) Re-enable `SparkSessionE2ESuite.interrupt tag`
[ https://issues.apache.org/jira/browse/SPARK-48139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48139: -- Affects Version/s: 3.5.2 > Re-enable `SparkSessionE2ESuite.interrupt tag` > -- > > Key: SPARK-48139 > URL: https://issues.apache.org/jira/browse/SPARK-48139 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0, 3.5.2 > Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Fix Version/s: 3.5.2 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness, pull-request-available > Fix For: 4.0.0, 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48183) Update error contribution guide to respect new error class file
[ https://issues.apache.org/jira/browse/SPARK-48183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48183. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46455 [https://github.com/apache/spark/pull/46455] > Update error contribution guide to respect new error class file > --- > > Key: SPARK-48183 > URL: https://issues.apache.org/jira/browse/SPARK-48183 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > We moved error class definition from .py to .json but documentation still > shows old behavior. We should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48183) Update error contribution guide to respect new error class file
[ https://issues.apache.org/jira/browse/SPARK-48183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48183: - Assignee: Haejoon Lee > Update error contribution guide to respect new error class file > --- > > Key: SPARK-48183 > URL: https://issues.apache.org/jira/browse/SPARK-48183 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Haejoon Lee >Assignee: Haejoon Lee >Priority: Major > Labels: pull-request-available > > We moved error class definition from .py to .json but documentation still > shows old behavior. We should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48152) Make spark-profiler as a part of release and publish to maven central repo
[ https://issues.apache.org/jira/browse/SPARK-48152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48152. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46402 [https://github.com/apache/spark/pull/46402] > Make spark-profiler as a part of release and publish to maven central repo > -- > > Key: SPARK-48152 > URL: https://issues.apache.org/jira/browse/SPARK-48152 > Project: Spark > Issue Type: Improvement > Components: Build, Documentation >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48152) Make spark-profiler as a part of release and publish to maven central repo
[ https://issues.apache.org/jira/browse/SPARK-48152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48152: - Assignee: BingKun Pan > Make spark-profiler as a part of release and publish to maven central repo > -- > > Key: SPARK-48152 > URL: https://issues.apache.org/jira/browse/SPARK-48152 > Project: Spark > Issue Type: Improvement > Components: Build, Documentation >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48178) Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed
[ https://issues.apache.org/jira/browse/SPARK-48178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48178. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46449 [https://github.com/apache/spark/pull/46449] > Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed > -- > > Key: SPARK-48178 > URL: https://issues.apache.org/jira/browse/SPARK-48178 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.5.2 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48178) Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed
[ https://issues.apache.org/jira/browse/SPARK-48178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48178: - Assignee: Dongjoon Hyun > Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed > -- > > Key: SPARK-48178 > URL: https://issues.apache.org/jira/browse/SPARK-48178 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.5.2 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48178) Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed
[ https://issues.apache.org/jira/browse/SPARK-48178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48178: -- Summary: Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed (was: Run `build/scala-211/java-11-17` jobs of branch-3.5 only if needed) > Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed > -- > > Key: SPARK-48178 > URL: https://issues.apache.org/jira/browse/SPARK-48178 > Project: Spark > Issue Type: Sub-task > Components: Project Infra >Affects Versions: 3.5.2 > Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48179) Pin `nbsphinx` to `0.9.3`
[ https://issues.apache.org/jira/browse/SPARK-48179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48179. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46448 [https://github.com/apache/spark/pull/46448] > Pin `nbsphinx` to `0.9.3` > -- > > Key: SPARK-48179 > URL: https://issues.apache.org/jira/browse/SPARK-48179 > Project: Spark > Issue Type: Bug > Components: Project Infra >Affects Versions: 3.5.2 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.5.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48179) Pin `nbsphinx` to `0.9.3`
[ https://issues.apache.org/jira/browse/SPARK-48179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48179: - Assignee: Dongjoon Hyun > Pin `nbsphinx` to `0.9.3` > -- > > Key: SPARK-48179 > URL: https://issues.apache.org/jira/browse/SPARK-48179 > Project: Spark > Issue Type: Bug > Components: Project Infra >Affects Versions: 3.5.2 > Reporter: Dongjoon Hyun > Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48179) Pin `nbsphinx` to `0.9.3`
Dongjoon Hyun created SPARK-48179: - Summary: Pin `nbsphinx` to `0.9.3` Key: SPARK-48179 URL: https://issues.apache.org/jira/browse/SPARK-48179 Project: Spark Issue Type: Bug Components: Project Infra Affects Versions: 3.5.2 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
Re: [DISCUSS] Spark 4.0.0 release
Thank you so much for the update, Wenchen! Dongjoon. On Tue, May 7, 2024 at 10:49 AM Wenchen Fan wrote: > UPDATE: > > Unfortunately, it took me quite some time to set up my laptop and get it > ready for the release process (docker desktop doesn't work anymore, my pgp > key is lost, etc.). I'll start the RC process at my tomorrow. Thanks for > your patience! > > Wenchen > > On Fri, May 3, 2024 at 7:47 AM yangjie01 wrote: > >> +1 >> >> >> >> *发件人**: *Jungtaek Lim >> *日期**: *2024年5月2日 星期四 10:21 >> *收件人**: *Holden Karau >> *抄送**: *Chao Sun , Xiao Li , >> Tathagata Das , Wenchen Fan < >> cloud0...@gmail.com>, Cheng Pan , Nicholas Chammas < >> nicholas.cham...@gmail.com>, Dongjoon Hyun , >> Cheng Pan , Spark dev list , >> Anish Shrigondekar >> *主题**: *Re: [DISCUSS] Spark 4.0.0 release >> >> >> >> +1 love to see it! >> >> >> >> On Thu, May 2, 2024 at 10:08 AM Holden Karau >> wrote: >> >> +1 :) yay previews >> >> >> >> On Wed, May 1, 2024 at 5:36 PM Chao Sun wrote: >> >> +1 >> >> >> >> On Wed, May 1, 2024 at 5:23 PM Xiao Li wrote: >> >> +1 for next Monday. >> >> >> >> We can do more previews when the other features are ready for preview. >> >> >> >> Tathagata Das 于2024年5月1日周三 08:46写道： >> >> Next week sounds great! Thank you Wenchen! >> >> >> >> On Wed, May 1, 2024 at 11:16 AM Wenchen Fan wrote: >> >> Yea I think a preview release won't hurt (without a branch cut). We don't >> need to wait for all the ongoing projects to be ready. How about we do a >> 4.0 preview release based on the current master branch next Monday? >> >> >> >> On Wed, May 1, 2024 at 11:06 PM Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >> Hey all, >> >> >> >> Reviving this thread, but Spark master has already accumulated a huge >> amount of changes. As a downstream project maintainer, I want to really >> start testing the new features and other breaking changes, and it's hard to >> do that without a Preview release. So the sooner we make a Preview release, >> the faster we can start getting feedback for fixing things for a great >> Spark 4.0 final release. >> >> >> >> So I urge the community to produce a Spark 4.0 Preview soon even if >> certain features targeting the Delta 4.0 release are still incomplete. >> >> >> >> Thanks! >> >> >> >> >> >> On Wed, Apr 17, 2024 at 8:35 AM Wenchen Fan wrote: >> >> Thank you all for the replies! >> >> >> >> To @Nicholas Chammas : Thanks for cleaning >> up the error terminology and documentation! I've merged the first PR and >> let's finish others before the 4.0 release. >> >> To @Dongjoon Hyun : Thanks for driving the >> ANSI on by default effort! Now the vote has passed, let's flip the config >> and finish the DataFrame error context feature before 4.0. >> >> To @Jungtaek Lim : Ack. We can treat the >> Streaming state store data source as completed for 4.0 then. >> >> To @Cheng Pan : Yea we definitely should have a >> preview release. Let's collect more feedback on the ongoing projects and >> then we can propose a date for the preview release. >> >> >> >> On Wed, Apr 17, 2024 at 1:22 PM Cheng Pan wrote: >> >> will we have preview release for 4.0.0 like we did for 2.0.0 and 3.0.0? >> >> Thanks, >> Cheng Pan >> >> >> > On Apr 15, 2024, at 09:58, Jungtaek Lim >> wrote: >> > >> > W.r.t. state data source - reader (SPARK-45511), there are several >> follow-up tickets, but we don't plan to address them soon. The current >> implementation is the final shape for Spark 4.0.0, unless there are demands >> on the follow-up tickets. >> > >> > We may want to check the plan for transformWithState - my understanding >> is that we want to release the feature to 4.0.0, but there are several >> remaining works to be done. While the tentative timeline for releasing is >> June 2024, what would be the tentative timeline for the RC cut? >> > (cc. Anish to add more context on the plan for transformWithState) >> > >> > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan >> wrote: >> > Hi all, >> > >> > It's close to the previously proposed 4.0.0 release date (June 2024), >> and I think it's time to prepare for it and
[jira] [Created] (SPARK-48178) Run `build/scala-211/java-11-17` jobs of branch-3.5 only if needed
Dongjoon Hyun created SPARK-48178: - Summary: Run `build/scala-211/java-11-17` jobs of branch-3.5 only if needed Key: SPARK-48178 URL: https://issues.apache.org/jira/browse/SPARK-48178 Project: Spark Issue Type: Sub-task Components: Project Infra Affects Versions: 3.5.2 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Upgrade `Parquet` to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Summary: Upgrade `Parquet` to 1.14.0 (was: Bump Parquet to 1.14.0) > Upgrade `Parquet` to 1.14.0 > --- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Affects Version/s: 4.0.0 (was: 3.5.2) > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48177: - Assignee: Fokko Driesprong > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48177) Bump Parquet to 1.14.0
[ https://issues.apache.org/jira/browse/SPARK-48177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48177: -- Fix Version/s: (was: 4.0.0) > Bump Parquet to 1.14.0 > -- > > Key: SPARK-48177 > URL: https://issues.apache.org/jira/browse/SPARK-48177 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: Fokko Driesprong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (ORC-1709) Upgrade GitHub Action `setup-java` to v4 and use built-in cache feature
Dongjoon Hyun created ORC-1709: -- Summary: Upgrade GitHub Action `setup-java` to v4 and use built-in cache feature Key: ORC-1709 URL: https://issues.apache.org/jira/browse/ORC-1709 Project: ORC Issue Type: Task Components: Infra Affects Versions: 2.1.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48037. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46273 [https://github.com/apache/spark/pull/46273] > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness, pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844388#comment-17844388 ] Dongjoon Hyun commented on SPARK-48037: --- Thank you, [~dzcxzl]. I raised the priority to `Blocker` for all future releases and added a label, `correctness`. > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48037: - Assignee: dzcxzl > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Affects Version/s: 3.4.3 3.5.1 4.0.0 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Target Version/s: 4.0.0, 3.5.2, 3.4.4 > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0, 4.0.0, 3.5.1, 3.4.3 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Labels: correctness (was: pull-request-available) > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: dzcxzl >Priority: Major > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48037) SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[ https://issues.apache.org/jira/browse/SPARK-48037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48037: -- Priority: Blocker (was: Major) > SortShuffleWriter lacks shuffle write related metrics resulting in > potentially inaccurate data > -- > > Key: SPARK-48037 > URL: https://issues.apache.org/jira/browse/SPARK-48037 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: dzcxzl >Priority: Blocker > Labels: correctness > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions
[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41547. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46432 [https://github.com/apache/spark/pull/46432] > Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions > -- > > Key: SPARK-41547 > URL: https://issues.apache.org/jira/browse/SPARK-41547 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Xinrong Meng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > See https://issues.apache.org/jira/browse/SPARK-41548 > We should fix the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48169) Use lazy BadRecordException cause for StaxXmlParser and JacksonParser
[ https://issues.apache.org/jira/browse/SPARK-48169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48169. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46438 [https://github.com/apache/spark/pull/46438] > Use lazy BadRecordException cause for StaxXmlParser and JacksonParser > - > > Key: SPARK-48169 > URL: https://issues.apache.org/jira/browse/SPARK-48169 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Vladimir Golubev >Assignee: Vladimir Golubev >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > For now since the https://issues.apache.org/jira/browse/SPARK-48143, the old > constructor is used -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48165) Update `ap-loader` to 3.0-9
[ https://issues.apache.org/jira/browse/SPARK-48165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48165. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46427 [https://github.com/apache/spark/pull/46427] > Update `ap-loader` to 3.0-9 > --- > > Key: SPARK-48165 > URL: https://issues.apache.org/jira/browse/SPARK-48165 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 4.0.0 >Reporter: BingKun Pan >Assignee: BingKun Pan >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48173) CheckAnalsis should see the entire query plan
[ https://issues.apache.org/jira/browse/SPARK-48173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48173. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46439 [https://github.com/apache/spark/pull/46439] > CheckAnalsis should see the entire query plan > - > > Key: SPARK-48173 > URL: https://issues.apache.org/jira/browse/SPARK-48173 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48173) CheckAnalsis should see the entire query plan
[ https://issues.apache.org/jira/browse/SPARK-48173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48173: - Assignee: Wenchen Fan > CheckAnalsis should see the entire query plan > - > > Key: SPARK-48173 > URL: https://issues.apache.org/jira/browse/SPARK-48173 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Wenchen Fan >Assignee: Wenchen Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48171) Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
[
https://issues.apache.org/jira/browse/SPARK-48171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-48171:
-
Assignee: Yang Jie
> Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
> -
>
> Key: SPARK-48171
> URL: https://issues.apache.org/jira/browse/SPARK-48171
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> /**
> * AbstractLogger constructor.
> *
> * Important: the log level set within
> * the {@link org.rocksdb.Options} instance will be used as
> * maximum log level of RocksDB.
> *
> * @param options {@link org.rocksdb.Options} instance.
> *
> * @deprecated Use {@link Logger#Logger(InfoLogLevel)} instead, e.g. {@code
> new
> * Logger(options.infoLogLevel())}.
> */
> @Deprecated
> public Logger(final Options options) {
> this(options.infoLogLevel());
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48171) Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
[
https://issues.apache.org/jira/browse/SPARK-48171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-48171.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 46436
[https://github.com/apache/spark/pull/46436]
> Clean up the use of deprecated APIs related to `o.rocksdb.Logger`
> -
>
> Key: SPARK-48171
> URL: https://issues.apache.org/jira/browse/SPARK-48171
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 4.0.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code:java}
> /**
> * AbstractLogger constructor.
> *
> * Important: the log level set within
> * the {@link org.rocksdb.Options} instance will be used as
> * maximum log level of RocksDB.
> *
> * @param options {@link org.rocksdb.Options} instance.
> *
> * @deprecated Use {@link Logger#Logger(InfoLogLevel)} instead, e.g. {@code
> new
> * Logger(options.infoLogLevel())}.
> */
> @Deprecated
> public Logger(final Options options) {
> this(options.infoLogLevel());
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48163) Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[
https://issues.apache.org/jira/browse/SPARK-48163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-48163.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 46425
[https://github.com/apache/spark/pull/46425]
> Disable `SparkConnectServiceSuite.SPARK-43923: commands send events -
> get_resources_command`
>
>
> Key: SPARK-48163
> URL: https://issues.apache.org/jira/browse/SPARK-48163
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
>Affects Versions: 4.0.0
> Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> {code}
> - SPARK-43923: commands send events ((get_resources_command {
> [info] }
> [info] ,None)) *** FAILED *** (35 milliseconds)
> [info] VerifyEvents.this.listener.executeHolder.isDefined was false
> (SparkConnectServiceSuite.scala:873)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48139) Re-enable `SparkSessionE2ESuite.interrupt tag`
[ https://issues.apache.org/jira/browse/SPARK-48139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48139: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Bug) > Re-enable `SparkSessionE2ESuite.interrupt tag` > -- > > Key: SPARK-48139 > URL: https://issues.apache.org/jira/browse/SPARK-48139 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48164: -- Component/s: Tests > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[ https://issues.apache.org/jira/browse/SPARK-48164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48164: -- Target Version/s: 4.0.0 > Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - > get_resources_command` > -- > > Key: SPARK-48164 > URL: https://issues.apache.org/jira/browse/SPARK-48164 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 4.0.0 > Reporter: Dongjoon Hyun >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48163) Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
[
https://issues.apache.org/jira/browse/SPARK-48163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-48163:
-
Assignee: Dongjoon Hyun
> Disable `SparkConnectServiceSuite.SPARK-43923: commands send events -
> get_resources_command`
>
>
> Key: SPARK-48163
> URL: https://issues.apache.org/jira/browse/SPARK-48163
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
>Affects Versions: 4.0.0
> Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
>
> {code}
> - SPARK-43923: commands send events ((get_resources_command {
> [info] }
> [info] ,None)) *** FAILED *** (35 milliseconds)
> [info] VerifyEvents.this.listener.executeHolder.isDefined was false
> (SparkConnectServiceSuite.scala:873)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48164) Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
Dongjoon Hyun created SPARK-48164: - Summary: Re-enable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command` Key: SPARK-48164 URL: https://issues.apache.org/jira/browse/SPARK-48164 Project: Spark Issue Type: Sub-task Components: Connect Affects Versions: 4.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48163) Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
Dongjoon Hyun created SPARK-48163:
-
Summary: Disable `SparkConnectServiceSuite.SPARK-43923: commands
send events - get_resources_command`
Key: SPARK-48163
URL: https://issues.apache.org/jira/browse/SPARK-48163
Project: Spark
Issue Type: Sub-task
Components: SQL, Tests
Affects Versions: 4.0.0
Reporter: Dongjoon Hyun
{code}
- SPARK-43923: commands send events ((get_resources_command {
[info] }
[info] ,None)) *** FAILED *** (35 milliseconds)
[info] VerifyEvents.this.listener.executeHolder.isDefined was false
(SparkConnectServiceSuite.scala:873)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
