Re: [Question] LimitedInputStream license issue in Spark source.

Since both license headers are Apache License 2.0, we don't see any issue there. They are compatible. The first line of the second license header means the file was copied from Google Guava project originally. Apache Spark community keeps the original header because it has `Authorship` part,

Re: [Question] LimitedInputStream license issue in Spark source.

May I ask why do you thinkn in that way? Could you elaborate a little more about your concerns if you mean it from a legal perspective? > The ASF header states "Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements.” > I ‘m not sure this is true with

Re: [ANNOUNCE] Apache Spark 3.4.0 released

Thank you, Xinrong! Dongjoon. On Fri, Apr 14, 2023 at 1:37 PM Xiao Li wrote: > Thank you Xinrong! > > Congratulations everyone! This is a great release with tons of new > features! > > > > Gengliang Wang 于2023年4月14日周五 13:04写道： > >> Congratulations everyone! >> Thank you Xinrong for driving

Re: [ANNOUNCE] Apache Spark 3.4.0 released

Apache Spark Docker images are published too. docker pull apache/spark:v3.4.0 docker pull apache/spark-py:v3.4.0 docker pull apache/spark-r:v3.4.0 Thanks, Dongjoon On Fri, Apr 14, 2023 at 2:56 PM Dongjoon Hyun wrote: > Thank you, Xinrong! > > Dongjoon. > > > On Fri, Apr 1

Re: hadoop-2 profile to be removed in 3.5.0

Thank you so much for head-ups, Chao! Dongjoon. On Fri, Apr 14, 2023 at 6:33 PM Chao Sun wrote: > Hi all, > > Just a heads up that `hadoop-2` profile is going to be removed in > Apache Spark 3.5.0. This has been discussed previously through this > email thread: >

Re: [ANNOUNCE] Apache Spark 3.4.0 released

Nice catch, Xiao! All `latest` tags are updated to v3.4.0 now. https://hub.docker.com/r/apache/spark/tags https://hub.docker.com/r/apache/spark-py/tags https://hub.docker.com/r/apache/spark-r/tags Dongjoon. On Fri, Apr 14, 2023 at 8:38 PM Xiao Li wrote: > @Dongjoon Hyun Thank

Re: Slack for Spark Community: Merging various threads

trolling can be possibly done via bots + designated >channel managers. >- allows channels like Slack for the organisation of messages. > > > On Mon, 10 Apr 2023 at 08:09, Dongjoon Hyun > wrote: > >> Thank you, Holden, Bjorn, Maciej. >> >> Yes, those are

Re: Spark 3.2.4 pom NOT FOUND on maven

Thank you for reporting, Enrico. I verified your issue report and also double-checked that both the original official Apache repository and Google Maven Mirror works correctly. Given that, it could be due to some transient issues because the artifacts are copied from Apache repository to Maven

Re: Slack for PySpark users

communities. TBH, we are kind of late. I think we can do the > same in our community? > > We can follow the guide when the ASF has an official process for ASF > archiving. Since our PMC are the owner of the slack workspace, we can make > a change based on the policy. WDYT? > >

Re: Slack for PySpark users

Hi, Xiao and all. (cc Matei) Please hold on the vote. There is a concern expressed by ASF board because recent Slack activities created an isolated silo outside of ASF mailing list archive. We need to establish a way to embrace it back to ASF archive before starting anything official. Bests,

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

+1 I also verified that RC5 has SBOM artifacts. https://repository.apache.org/content/repositories/orgapachespark-1439/org/apache/spark/spark-core_2.12/3.4.0/spark-core_2.12-3.4.0-cyclonedx.json

Re: Slack for PySpark users

rom relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 3 Apr 2023 at 20:59, Dongjoon Hyun > wrote: > >> As Mich Tal

Re: Slack for PySpark users

, I stand >>>>>> corrected >>>>>> - To be clear, I intentionally didn't refer to any specific mailing >>>>>> list because we didn't set up any rule here yet. >>>>>>fair enough >>>>>> >>>>>>

Re: Slack for PySpark users

to keep this > active. > > > > On Mon, Apr 3, 2023 at 16:46 Dongjoon Hyun > wrote: > >> Shall we summarize the discussion so far? >> >> To sum up, "ASF Slack" vs "3rd-party Slack" was the real background to >> initiate this thread instead

Re: Slack for PySpark users

reference. They are going with the way they are convenient. >>> >>> Same applies here - if ASF Slack requires a restricted invitation >>> mechanism then it won't work. Looks like there is a link for an invitation, >>> but we are also talking about the cost as well

Apache Spark 3.2.4 EOL Release?

Hi, All. Since Apache Spark 3.2.0 passed RC7 vote on October 12, 2021, branch-3.2 has been maintained and served well until now. - https://github.com/apache/spark/releases/tag/v3.2.0 (tagged on Oct 6, 2021) - https://lists.apache.org/thread/jslhkh9sb5czvdsn7nz4t40xoyvznlc7 As of today,

Re: [VOTE] Release Apache Spark 3.4.0 (RC6)

Thank you! Dongjoon On Fri, Apr 7, 2023 at 2:16 PM Xinrong Meng wrote: > I am able to proceed with the release now. I'll send an announcement when > the RC cut is completed. > > Xinrong > > On Fri, Apr 7, 2023 at 9:54 AM Dongjoon Hyun > wrote: > >> Got it. Tha

Re: [VOTE] Release Apache Spark 3.4.0 (RC6)

Got it. Thank you for sharing the current status. Dongjoon. On Fri, Apr 7, 2023 at 9:21 AM Xinrong Meng wrote: > Hi Dongjoon, > > Yes, it is. To be more specific, we failed to build documentation for RC7 > because of the sbt build outage. > > Xinrong > > On Fri, Apr 7,

Re: Apache Spark 3.2.4 EOL Release?

Thank you for reporting. I'll check that, too. Dongjoon. On Thu, Apr 6, 2023 at 1:13 AM yangjie01 wrote: > Hi, Dongjoon Hyun > > Maybe we need include the fix of SPARK-39696 in Apache Spark 3.2.4 EOL > Release, this will fix a data race issue in access to > TaskMetrics.exte

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 I verified the same steps like previous RCs. Dongjoon. On Sat, Apr 8, 2023 at 7:47 PM Mridul Muralidharan wrote: > > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > Regards, > Mridul > > > On Sat, Apr 8, 2023

[VOTE] Release Apache Spark 3.2.4 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.2.4. The vote is open until April 13th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.2.4 [ ] -1 Do not release this package because

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

Oh, there is a typo in the mail. The following should be `April` instead of `August`. > August 13th 1AM (PST) Dongjoon. On 2023/04/09 23:38:00 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.2.4. > > The vote is open until

[VOTE] Release Apache Spark 3.2.4 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.2.4. The vote is open until August 13th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.2.4 [ ] -1 Do not release this package because

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

I'll start with my +1. I verified the checksum, signatures of the artifacts, and documentations. Also, ran the tests with YARN and K8s modules. Dongjoon. On 2023/04/09 23:46:10 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.2.4. > &

Re: Slack for Spark Community: Merging various threads

ghting, > and code playgrounds <https://zulip.com/help/code-blocks#code-playgrounds> > . > > > > > > > fre. 7. apr. 2023 kl. 18:54 skrev Holden Karau : > >> I think there was some concern around how to make any sync channel show >> up in logs / index

Re: sbt build is broken because repo is not available

Thank you for the pointer, Yuming. Dongjoon. On Fri, Apr 7, 2023 at 12:18 AM Yuming Wang wrote: > Hi all, > > sbt build is broken because repo is not available. Please see: > https://github.com/sbt/sbt/issues/7202. > >

Re: [VOTE] Release Apache Spark 3.4.0 (RC6)

Hi, Xinrong. I saw the RC7 tag. Maybe, RC7 vote is blocked due to the on-going build outage? Dongjoon. On Thu, Apr 6, 2023 at 6:17 PM Xinrong Meng wrote: > Thank you! Let me recut the RC then. > > On Thu, Apr 6, 2023 at 6:14 PM Hyukjin Kwon wrote: > >> Merged the fix. >> >> On Fri, 7 Apr

Re: Slack for Spark Community: Merging various threads

Thank you, All. I'm very satisfied with the focused and right questions for the real issues by removing irrelevant claims. :) Let me collect your relevant comments simply. # Category 1: Invitation Hurdle > The key question here is that do PMC members have the bandwidth of inviting everyone in

Re: Apache Spark 3.2.4 EOL Release?

Thank you all. Dongjoon. On 2023/04/05 18:32:07 Gengliang Wang wrote: > +1 > > On Wed, Apr 5, 2023 at 11:27 AM kazuyuki tanimura > wrote: > > > +1 > > > > On Apr 5, 2023, at 6:53 AM, Tom Graves > > wrote: > > > > +1 > > > > Tom

Re: Slack for Spark Community: Merging various threads

Thank you so much, Denny. Yes, let me comment on a few things. > - While there is an ASF Slack , it >requires an @apache.org email address 1. This sounds a little misleading because we can see `guest` accounts in the same link. People can be invited by

[VOTE][RESULT] Release Spark 3.2.4 (RC1)

The vote passes with 16 +1s (7 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Liang-Chi Hsieh * - Kent Yao - Sean Owen * - Yang Jie - Chao Sun - Huaxin Gao * - Mridul Muralidharen * - Yuming Wang - Ruifeng Zheng - Hyukjin Kwon * - Wenchen Fan

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

>>> Ruifeng Zheng > > > >>>> ruife...@foxmail.com > > > >>>> > > > >>>> < > > https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage=true=Ruifeng++Zheng=https%3A%2F%2Fthirdqq.qlogo.cn%2Fg%3Fb%3Doidb%26k%3DTf

Re: ASF board report draft for Feb 2023

Thank you, Matei. Could you include the following addtionally? 1. Liang-Chi is preparing v3.3.2 (This month). https://lists.apache.org/thread/nwzr3o2cxyyf6sbb37b8yylgcvmbtp16 2. Since Spark 3.4.0, we attached SBOM to Apache Spark Maven artifacts [SPARK-41893] in line with other ASF projects.

Re: [VOTE] Release Spark 3.3.2 (RC1)

Hi, All. As the author of that `Improvement` patch, I strongly disagree with giving the wrong idea which Python 3.11 is officially supported in Spark 3.3. I only developed and delivered it for Apache Spark 3.4.0 specifically as `Improvement`. We may want to backport it branch-3.3 but it's also

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

+1 Dongjoon On 2023/02/13 22:52:59 "L. C. Hsieh" wrote: > Hi all, > > I'd like to start the vote for SPIP: Lazy Materialization for Parquet > Read Performance Improvement. > > The high summary of the SPIP is that it proposes an improvement to the > Parquet reader with lazy materialization

Re: [VOTE] Release Spark 3.3.2 (RC1)

+1 I also verified additional internal tests. Dongjoon. On Sat, Feb 11, 2023 at 11:17 AM Mridul Muralidharan wrote: > > Looks like it was an issue with wget not fetching all the artifacts, my > bad ! > > Looks good to me, +1 for release - thanks ! > > > Regards, > Mridul > > > On Sat, Feb 11,

Re: [VOTE][RESULT] Release Spark 3.3.2 (RC1)

Great! Thank you, Liang-Chi! Dongjoon. On Wed, Feb 15, 2023 at 9:22 AM L. C. Hsieh wrote: > The vote passes with 12 +1s (4 binding +1s). > Thanks to all who helped with the release! > > (* = binding) > +1: > - Mridul Muralidharan (*) > - Dongjoon Hyun (*) > - Sean Ow

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

+1 On Wed, Feb 1, 2023 at 12:52 AM Mich Talebzadeh wrote: > +1 > > > >view my Linkedin profile > > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any >

Re: [DISCUSS] Make release cadence predictable

+1 for Hyukjin and Sean's opinion. Thank you for initiating this discussion. If we have a fixed-predefined regular 6-month, I believe we can persuade the incomplete features to wait for next releases more easily. In addition, I want to add the first RC1 date requirement because RC1 always did a

[ANNOUNCE] Apache Spark 3.2.4 released

. Dongjoon Hyun

Re: Gauging interest in: ScalaFix + Scala Steward for Spark 4.0

([VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)) Holden, could you think it in this way too? Thanks, Dongjoon. On 2023/06/12 18:57:32 Holden Karau wrote: > Yup I think buidling consensus on what goes in 4.X is something we’ll need > to do. > > On Mon, Jun 12, 2023 at 11:56 AM Do

Re: Gauging interest in: ScalaFix + Scala Steward for Spark 4.0

Thank you for sharing those. I'm also interested in taking advantage of it. Also, I hope `spark-upgrade` can help us in line with Spark 4.0. However, we don't need to discuss any of this if we don't build a consensus on both Spark 4.0 or next Scala version. We don't have a vehicle at all to

Re: Spark 3.5 Branch Cut

Thank you so much, Yuanjian! Dongjoon. On Mon, Jul 17, 2023 at 1:05 PM Yuanjian Li wrote: > Hi, all > > FYI, I cut branch-3.5 as https://github.com/apache/spark/tree/branch-3.5 > > Here is the complete list of exception merge requests received before the > cut: > >- > >SPARK-44421:

Re: Spark Docker Official Image is now available

Thank you! Dongjoon On Thu, Jul 20, 2023 at 8:40 AM Xiao Li wrote: > Thank you, Yikun! This is great! > > On Wed, Jul 19, 2023 at 7:55 PM Ruifeng Zheng wrote: > >> Awesome, thank you YiKun for driving this! >> >> On Thu, Jul 20, 2023 at 9:12 AM Hyukjin Kwon >> wrote: >> >>> This is amazing,

Re: Time for Spark v3.5.0 release

+1 Thank you, Yuanjian Dongjoon On Tue, Jul 4, 2023 at 1:03 AM Hyukjin Kwon wrote: > Yeah one day postponed shouldn't be a big deal. > > On Tue, Jul 4, 2023 at 7:10 AM Yuanjian Li wrote: > >> Hi All, >> >> According to the Spark versioning policy at >>

Apache Spark 3.5.0 Expectations (?)

Hi, All. Apache Spark 3.5.0 is scheduled for August (1st Release Candidate) and currently a few notable things are under discussions in the mailing list. I believe it's a good time to share a short summary list (containing both completed and in-progress items) to give a highlight in advance and

Apache Spark 4.0 Timeframe?

Hi, All. I'd like to propose to start to prepare Apache Spark 4.0 after creating branch-3.5 on July 16th. - https://spark.apache.org/versioning-policy.html Historically, the Apache Spark release dates have the following timeframes and we already have Spark 3.5 plan which will be maintained up

Re: Apache Spark 3.5.0 Expectations (?)

Thank you all for your replies. 1. Thank you, Jia, for those JIRAs. 2. Sounds great for "Scala 2.13 for Spark 4.0". I'll initiate a new thread for that. - "I wonder if it’s safer to do it in Spark 4 (which I believe will be discussed soon)." - "I would make it the default at 4.0, myself."

Re: [DISCUSS] Add SQL functions into Scala, Python and R API

Thank you for the proposal. I'm wondering if we are going to consider them as release blockers or not. In general, I don't think those SQL functions should be available in all languages as release blockers. (Especially in R or new Spark Connect languages like Go and Rust). If they are not

ASF policy violation and Scala version issues

Hi, All and Matei (as the Chair of Apache Spark PMC). Sorry for a long email, I want to share two topics and corresponding action items. You can go to "Section 3: Action Items" directly for the conclusion. ### 1. ASF Policy Violation ### ASF has a rule for "MAY I CALL MY MODIFIED CODE

Re: Apache Spark 4.0 Timeframe?

ing the Scala version. > > > > But I want to know if Spark 4.0 chooses to use the Scala 2.13.x, is it > impossible to switch Scala 3.x as the default version during the lifecycle > of Spark 4.x? > > > > Thanks > > Yang Jie > > > > *发件人**: *Dongjoon Hyun >

Apache Spark 3.4.1 Release?

Hi, All. `branch-3.4` already has 77 commits since v3.4.0 tag. https://github.com/apache/spark/releases/v3.4.0 (Tagged on April 6th) $ git log --oneline v3.4.0..HEAD | wc -l 77 I'd like to propose to have Apache Spark 3.4.1 before DATA+AI Summit (June 26~29) because that

Re: ASF policy violation and Scala version issues

It goes to "legal-discuss@". https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4 I hope we can conclude the legal part clearly and shortly in one way or another which we will follow with confidence. Dongjoon On 2023/06/06 20:06:42 Dongjoon Hyun wrote: > Thank yo

Re: ASF policy violation and Scala version issues

Thank you, Sean, Mich, Holden, again. For this specific part, let's ask the ASF board via bo...@apache.org to find a right answer because it's a controversial legal issue here. > I think you'd just prefer Databricks make a different choice, which is legitimate, but, an issue to take up with

Re: ASF policy violation and Scala version issues

ce. Dongjoon. On Tue, Jun 6, 2023 at 2:49 PM Dongjoon Hyun wrote: > It goes to "legal-discuss@". > > https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4 > > I hope we can conclude the legal part clearly and shortly in one way or > another which we

Re: ASF policy violation and Scala version issues

like an issue, even desirable. > > 2d/ Same as 2b > > 3/ I don't think 1/ is an incident. Yes to moving towards 4.0 after 3.5, > IMHO, and to removing Ammonite in 4.0 if there is no resolution forthcoming > > On Mon, Jun 5, 2023 at 2:46 AM Dongjoon Hyun > wrote: > >> Hi, All and Mate

Re: JDK version support policy?

I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too. Dongjoon. On 2023/06/07 02:42:19 yangjie01 wrote: > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only > support Java 17 and the upcoming Java 21. > > 发件人: Denny Lee > 日期: 2023年6月7日 星期三 07:10 > 收件人: Sean

Re: ASF policy violation and Scala version issues

uot; > - UI shows Apache Spark logo and `3.4.0`. Dongjoon. On Mon, Jun 5, 2023 at 10:40 AM Sean Owen wrote: > On Mon, Jun 5, 2023 at 12:01 PM Dongjoon Hyun > wrote: > >> 1. For the naming, yes, but the company should use different version >> numbers instead of the e

Re: ASF policy violation and Scala version issues

Sean, it seems that you are confused here. We are not talking about your upper system (the notebook environment). We are talking about the submodule, "Apache Spark 3.4.0-databricks". Whatever you call it, both of us knows "Apache Spark 3.4.0-databricks" is different from "Apache Spark 3.4.0".

Re: ASF policy violation and Scala version issues

suggestion, > and I think it does nothing in particular for users. You've made the > suggestion, and I do not see some police action from the PMC must follow. > > > I think you're simply objecting to a vendor choice, but that is not > on-topic here unless you can specifically rebut the reas

Re: ASF policy violation and Scala version issues

ed that AWS EMR does exactly the same thing. >>> We choose the EMR version (e.g., 6.4.0) and it has an associated Spark >>> version (e.g., 3.1.2). >>> The Spark version here is not the original Apache version but AWS Spark >>> distribution. >>> >>>

Re: ASF policy violation and Scala version issues

tions > to unblock this situation in a long-term maintenance perspective. > - Replace it with a Scala-shell based implementation > - Move `connector/connect/client/jvm/pom.xml` outside from Spark repo. >Maybe, we can put it into the new repo like Rust and Go client. &

Re: Apache Spark 3.4.1 Release?

Thank you all. I'll check and prepare `branch-3.4` for the target date, June 20th. Dongjoon. On Fri, Jun 9, 2023 at 10:47 PM yangjie01 wrote: > +1 > > > > Thank you Dongjoon ~ > > > > *发件人**: *Ruifeng Zheng > *日期**: *2023年6月10日 星期六 09:39 > *收件人**: *Xiao Li > *抄送**: *Wenchen Fan , Xinrong

Re: ASF policy violation and Scala version issues

Let me add my answers about a few Scala questions, Jungtaek. > Are we concerned that a library does not release a new version > which bumps the Scala version, which the Scala version is > announced in less than a week? No, we have concerns about the newly introduced disability in the Apache

Re: [CONNECT] New Clients for Go and Rust

+1 for starting on a separate repo. Dongjoon. On Thu, May 25, 2023 at 9:53 AM yangjie01 wrote: > +1 on start this with a separate repo. > > Which new clients can be placed in the main repo should be discussed after > they are mature enough, > > > > Yang Jie > > > > *发件人**: *Denny Lee > *日期**:

Re: [Reminder] Spark 3.5 RC Cut

fact that 3.3.6 is also affected. >> >> > HADOOP-18757 seems to be merged just two weeks ago and there is no >> > Apache Hadoop release with it, isn't it? >> >> That is correct, there is no hadoop release containing the fix. So >> therefore 3.3.6 would als

Re: [Reminder] Spark 3.5 RC Cut

Hi, Emil. HADOOP-18568 is still open and it seems to be never a part of the Hadoop trunk branch. Do you mean another JIRA? Dongjoon. On Tue, Aug 1, 2023 at 2:59 AM Emil Ejbyfeldt wrote: > Hi, > > We previously ran some experiments on builds from the 3.5 branch and > noticed that Hadoop had

Re: Spark 3.0.0 EOL

As Hyukjin replied, Apache Spark 3.0.0 is already in EOL status. To Pralabh, FYI, in the community, - Apache Spark 3.2 also reached the EOL already. https://lists.apache.org/thread/n4mdfwr5ksgpmrz0jpqp335qpvormos1 If you are considering Apache Spark 4, here is the other 3.x timeline, -

Re: Time for Spark 3.3.3 release?

+1 Thank you for volunteering, Yuming. Dongjoon On Fri, Jul 28, 2023 at 11:35 AM Yuming Wang wrote: > Hi Spark devs, > > Since Apache Spark 3.3.2 tag creation (Feb 11), 60 patches > have > arrived at branch-3.3. > > Shall we make

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Dongjoon On 2023/08/10 07:14:07 yangjie01 wrote: > +1 > Thanks, Jie Yang > > > 发件人: Yuming Wang > 日期: 2023年8月10日 星期四 13:33 > 收件人: Dongjoon Hyun > 抄送: dev > 主题: Re: [VOTE] Release Apache Spark 3.3.3 (RC1) > > +1 myself. > > On Tue

Re: KubernetesLocalDiskShuffleDataIO mount path dependency doubt.

Hi, Arun. SPARK-35593 (Support shuffle data recovery on the reused PVCs) was Apache Spark 3.2.0 feature whose plugin follows only the legacy Spark shuffle directory structure to be safe. You can see the AS-IS test coverage in the corresponding `KubernetesLocalDiskShuffleDataIOSuite`.

Re: [Reminder] Spark 3.5 RC Cut

Thank you again, Emil and Bjorn. FYI, SPARK-44678 landed at branch-3.5 like the following. https://github.com/apache/spark/pull/42345 [SPARK-44678][BUILD][3.5] Downgrade Hadoop to 3.3.4 Dongjoon. On 2023/08/02 18:58:51 Bjørn Jørgensen wrote: > @Dongjoon Hyun FYI > [image: image.png] &

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

Hi, Yuming. One of the community GitHub Action test pipelines is unhealthy consistently due to Python mypy linter. https://github.com/apache/spark/actions/workflows/build_branch33.yml It seems due to the pipeline difference between the same Python mypy linter already pass in commit build,

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

st and the master use the same yml file. > > > > Jie Yang > > > > *发件人**: *Dongjoon Hyun > *日期**: *2023年8月8日 星期二 00:18 > *收件人**: *Yuming Wang > *抄送**: *dev > *主题**: *Re: [VOTE] Release Apache Spark 3.3.3 (RC1) > > > > Hi, Yuming. > > > >

Re: Welcome two new Apache Spark committers

Congratulations, Peter and Xiduo. :) Dongjoon. On Sun, Aug 6, 2023 at 10:08 PM XiDuo You wrote: > Thank you all ! > > Jia Fan 于2023年8月7日周一 11:31写道： > > > > Congratulations! > > > > > > Jia Fan > > > > > > 2023年8月7日 11:28，Ye Xianjin 写道： > > > > Congratulations! > > >

Re: ASF board report draft for August 2023

Thank you, Matei. It looks good to me. Dongjoon On Mon, Aug 7, 2023 at 22:54 Matei Zaharia wrote: > It’s time to send our quarterly report to the ASF board on August 9th. > Here’s what I wrote as a draft — feel free to suggest changes. > > = > > Issues for the

Re: [Reminder] Spark 3.5 RC Cut

more, please? Dongjoon. On Tue, Aug 1, 2023 at 9:46 PM Emil Ejbyfeldt wrote: > Hi, > > Yes, sorry about that seem to have messed up the link. Should have been > https://issues.apache.org/jira/browse/HADOOP-18757 > > Best, > Emil > > On 01/08/2023 19:08, Dongjo

Re: [VOTE][RESULT] Release Plan for Apache Spark 4.0.0 (June 2024)

be >> fixed in pandas 3.5.0. >> >> *To recap, all breaking changes related to pandas 2.0.0 will be supported >> in Spark 4.0.0,* *and will remain deprecated with appropriate errors in >> Spark 3.5.0.* >> >> >> >> https://issues.apache.org/jira/browse/SP

Apache Spark 4.0.0 Dev Item Planning (SPARK-44111)

Hi, All. As a continuation of our previous discussion, the official Apache Spark 4.0 Plan JIRA is created today in order to collect the community dev items. Feel free to add your work items, ideas, suggestions, aspirations and interests. We will moderate together.

Re: [VOTE] Release Spark 3.4.1 (RC1)

+1 Dongjoon On 2023/06/20 02:51:32 Jia Fan wrote: > +1 > > Dongjoon Hyun 于2023年6月20日周二 10:41写道： > > > Please vote on releasing the following candidate as Apache Spark version > > 3.4.1. > > > > The vote is open until June 23rd 1AM (PST) and passes if

Re: Apache Spark 4.0.0 Dev Item Planning (SPARK-44111)

ion. > SPARK-38506 <https://issues.apache.org/jira/browse/SPARK-38506> Push > partial aggregation through join > > > On Wed, Jun 21, 2023 at 4:42 AM Dongjoon Hyun wrote: > >> Hi, All. >> >> As a continuation of our previous discussion, the official Apac

Re: [VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

say everyone should *and* http UA in all the clients who make calls of > object stores should, as it helps field issues there. s3a and abfs clients > do provide the ability to add params there -please set them in your > deployments > > On Fri, 16 Jun 2023 at 21:53, Dongjoon Hyun wrote

Re: [VOTE] Release Spark 3.4.1 (RC1)

gt; >> Pozdrawiam, > >> Jacek Laskowski > >> > >> "The Internals Of" Online Books > >> Follow me on https://twitter.com/jaceklaskowski > >> > >> > >> > >> On Tue, Jun 20, 2023 at 4:41 AM Dongjoon Hy

[ANNOUNCE] Apache Spark 3.4.1 released

. Dongjoon Hyun

[VOTE][RESULT] Release Spark 3.4.1 (RC1)

The vote passes with 15 +1s (10 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Jia Fan - Dongjoon Hyun * - Liang-Chi Hsieh * - Yang Jie - Hyukjin Kwon * - Huaxin Gao * - Ruifeng Zheng * - Peter Toth - Xinrong Meng * - Jacek Laskowski - Yuming Wang * - Chao Sun

[VOTE][RESULT] Apache Spark PMC asks Databricks to differentiate its Spark version string

The vote failed with one +1 (binding), one +0 (binding), and three -1s (two binding -1s). Thanks to all for your participation. (* = binding) +1: - Dongjoon Hyun * +0: None - Maciej Szymkiewicz * -1: None - Sean Owen * - Hyukjin Kwon * - Mich Talebzadeh

[VOTE] Release Spark 3.4.1 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.4.1. The vote is open until June 23rd 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.1 [ ] -1 Do not release this package because

Re: [VOTE][RESULT] Release Plan for Apache Spark 4.0.0 (June 2024)

g that, we can - hopefully - > have a grounded discussion. > > Cheers, > Herman > > On Mon, Jun 19, 2023 at 4:01 PM Dongjoon Hyun wrote: > >> Thank you. I reviewed the threads, vote and result once more. >> >> I found that I missed the binding vote mark on Ho

Re: [VOTE][RESULT] Release Plan for Apache Spark 4.0.0 (June 2024)

rst is the > opposite order procedurally. > The vote passed as a procedural issue, but I would prefer to consider this > as a tentative date, and should probably need another vote to adjust the > date considering the plans, preview dates, and items we aim for 4.0.0. >

Re: Apache Spark 4.0 Timeframe?

re your opinion for Apache Spark 4.0 here? Best, Dongjoon. On Wed, May 31, 2023 at 6:02 PM Dongjoon Hyun wrote: > Hi, All. > > I'd like to propose to start to prepare Apache Spark 4.0 after creating > branch-3.5 on July 16th. > > - https://spark.apache.org/versioning-policy.

Re: [VOTE][RESULT] Release Plan for Apache Spark 4.0.0 (June 2024)

including the following - Apache Spark 4.0.0 Preview (and Dates) - Apache Spark 4.0.0 Items - Apache Spark 4.0.0 Plan Adjustment Please initiate the discussion. Thanks, Dongjoon. On 2023/06/16 19:30:42 Dongjoon Hyun wrote: > The vote passes with 6 +1s (4 binding +1s), one -0, and one -1. >

[VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

Please vote on the following statement. The vote is open until June 23th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. Apache Spark PMC asks Databricks to differentiate its Spark version string to avoid confusions because Apache Spark PMC is responsible

[VOTE][RESULT] Release Plan for Apache Spark 4.0.0 (June 2024)

The vote passes with 6 +1s (4 binding +1s), one -0, and one -1. Thank you all for your participation and especially your additional comments during this voting, Mridul, Hyukjin, and Jungtaek. (* = binding) +1: - Dongjoon Hyun * - Huaxin Gao * - Liang-Chi Hsieh * - Kazuyuki Tanimura - Chao Sun

Re: ASF policy violation and Scala version issues

a consensus and have a conclusion. Dongjoon On 2023/06/12 08:15:39 Dongjoon Hyun wrote: > Let me add my answers about a few Scala questions, Jungtaek. > > > Are we concerned that a library does not release a new version > > which bumps the Scala version, which the Scala versio

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

t;> >>>>>>> >>>>>>> On Jun 12, 2023, at 11:32 AM, Holden Karau >>>>>>> wrote: >>>>>>> >>>>>>> -0 >>>>>>> >>>>>>> I'd like to

Re: [VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

+1 Dongjoon On 2023/06/16 19:53:03 Dongjoon Hyun wrote: > Please vote on the following statement. The vote is open until June 23th > 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of > 3 +1 votes. > > Apache Spark PMC asks Databricks to differen

Re: [VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

No, this is a vote on dev@ intentionally as a part of our previous thread, "ASF policy violation and Scala version issues" ( https://lists.apache.org/thread/k7gr65wt0fwtldc7hp7bd0vkg1k93rrb) > did you mean this for the PMC list? I clearly started the thread with the following. > - Apache Spark

Re: [VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

the board report, for sure. We do not include > invalid issues in the board report. That part needs no decision from anyone. > > > On Fri, Jun 16, 2023 at 3:08 PM Dongjoon Hyun > wrote: > >> No, this is a vote on dev@ intentionally as a part of our previous >>

[VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

Please vote on the release plan for Apache Spark 4.0.0. The vote is open until June 16th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Have a release plan for Apache Spark 4.0.0 (June 2024) [ ] -1 Do not have a plan for Apache Spark 4.0.0 because

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

+1 Dongjoon On 2023/06/12 18:00:38 Dongjoon Hyun wrote: > Please vote on the release plan for Apache Spark 4.0.0. > > The vote is open until June 16th 1AM (PST) and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Have a release plan for

<    2   3   4   5   6   7   8   >