Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Prashant Sharma
Hi Folks, So, I am back, and searched the JIRAS with target version as "2.4.7" and Resolved, found only 2 jiras. So, are we good to go, with just a couple of jiras fixed ? Shall I proceed with making a RC? Thanks, Prashant On Thu, Jul 2, 2020 at 5:23 PM Prashant Sharma wrote: > Thank you,

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Nick Pentreath
Congratulations and welcome as Apache Spark committers! On Wed, 15 Jul 2020 at 06:59, Prashant Sharma wrote: > Congratulations all ! It's great to have such committed folks as > committers. :) > > On Wed, Jul 15, 2020 at 9:24 AM Yi Wu wrote: > >> Congrats!! >> >> On Wed, Jul 15, 2020 at 8:02

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Prashant Sharma
Congratulations all ! It's great to have such committed folks as committers. :) On Wed, Jul 15, 2020 at 9:24 AM Yi Wu wrote: > Congrats!! > > On Wed, Jul 15, 2020 at 8:02 AM Hyukjin Kwon wrote: > >> Congrats! >> >> 2020년 7월 15일 (수) 오전 7:56, Takeshi Yamamuro 님이 작성: >> >>> Congrats, all! >>> >>>

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Yi Wu
Congrats!! On Wed, Jul 15, 2020 at 8:02 AM Hyukjin Kwon wrote: > Congrats! > > 2020년 7월 15일 (수) 오전 7:56, Takeshi Yamamuro 님이 작성: > >> Congrats, all! >> >> On Wed, Jul 15, 2020 at 5:15 AM Takuya UESHIN >> wrote: >> >>> Congrats and welcome! >>> >>> On Tue, Jul 14, 2020 at 1:07 PM Bryan Cutler

Re: [PSA] Apache Spark uses GitHub Actions to run the tests

2020-07-14 Thread Holden Karau
This has already helped me catch a potential flaky test I might have otherwise missed. So excited for this additional signal :) On Tue, Jul 14, 2020 at 6:19 PM Takeshi Yamamuro wrote: > Thanks, Hyukjin! > > > Therefore, I do believe PRs can be merged in most general cases once the > Jenkins PR

Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Yi Wu
Thanks Dongjoon, you're right. I referenced to a wrong caused-by jira link in https://issues.apache.org/jira/browse/SPARK-32307. Bests, Yi On Wed, Jul 15, 2020 at 3:03 AM Dongjoon Hyun wrote: > Hi, Yi. > > Could you explain why you think that is a blocker? For the given example > from the JIRA

Re: [PSA] Apache Spark uses GitHub Actions to run the tests

2020-07-14 Thread Takeshi Yamamuro
Thanks, Hyukjin! > Therefore, I do believe PRs can be merged in most general cases once the Jenkins PR builder or Github Actions build passes greatly helpful! Bests, Takeshi On Tue, Jul 14, 2020 at 4:14 PM Hyukjin Kwon wrote: > Perfect. Plus, Github Actions is only for master branch at this

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Hyukjin Kwon
Congrats! 2020년 7월 15일 (수) 오전 7:56, Takeshi Yamamuro 님이 작성: > Congrats, all! > > On Wed, Jul 15, 2020 at 5:15 AM Takuya UESHIN > wrote: > >> Congrats and welcome! >> >> On Tue, Jul 14, 2020 at 1:07 PM Bryan Cutler wrote: >> >>> Congratulations and welcome! >>> >>> On Tue, Jul 14, 2020 at 12:36

Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Takeshi Yamamuro
> Just wanted to check if there are any blockers that we are still waiting for to start the new release process. I don't see any on-going blocker in my area. Thanks for the notification. Bests, Tkaeshi On Wed, Jul 15, 2020 at 4:03 AM Dongjoon Hyun wrote: > Hi, Yi. > > Could you explain why you

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Takeshi Yamamuro
Congrats, all! On Wed, Jul 15, 2020 at 5:15 AM Takuya UESHIN wrote: > Congrats and welcome! > > On Tue, Jul 14, 2020 at 1:07 PM Bryan Cutler wrote: > >> Congratulations and welcome! >> >> On Tue, Jul 14, 2020 at 12:36 PM Xingbo Jiang >> wrote: >> >>> Welcome, Huaxin, Jungtaek, and Dilip! >>>

Re: [DISCUSS] Drop Python 2, 3.4 and 3.5

2020-07-14 Thread Holden Karau
I’m going to drink a celebratory afternoon coffee :) On Tue, Jul 14, 2020 at 12:26 PM shane knapp ☠ wrote: > this is seriously great news! let's all take a moment and welcome apache > spark's python support to the present. ;) > > On Mon, Jul 13, 2020 at 7:26 PM Holden Karau wrote: > >>

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Takuya UESHIN
Congrats and welcome! On Tue, Jul 14, 2020 at 1:07 PM Bryan Cutler wrote: > Congratulations and welcome! > > On Tue, Jul 14, 2020 at 12:36 PM Xingbo Jiang > wrote: > >> Welcome, Huaxin, Jungtaek, and Dilip! >> >> Congratulations! >> >> On Tue, Jul 14, 2020 at 10:37 AM Matei Zaharia >> wrote:

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Bryan Cutler
Congratulations and welcome! On Tue, Jul 14, 2020 at 12:36 PM Xingbo Jiang wrote: > Welcome, Huaxin, Jungtaek, and Dilip! > > Congratulations! > > On Tue, Jul 14, 2020 at 10:37 AM Matei Zaharia > wrote: > >> Hi all, >> >> The Spark PMC recently voted to add several new committers. Please join

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Xingbo Jiang
Welcome, Huaxin, Jungtaek, and Dilip! Congratulations! On Tue, Jul 14, 2020 at 10:37 AM Matei Zaharia wrote: > Hi all, > > The Spark PMC recently voted to add several new committers. Please join me > in welcoming them to their new roles! The new committers are: > > - Huaxin Gao > - Jungtaek

Re: [DISCUSS] Drop Python 2, 3.4 and 3.5

2020-07-14 Thread shane knapp ☠
this is seriously great news! let's all take a moment and welcome apache spark's python support to the present. ;) On Mon, Jul 13, 2020 at 7:26 PM Holden Karau wrote: > Awesome, thanks you for driving this forward :) > > On Mon, Jul 13, 2020 at 7:25 PM Hyukjin Kwon wrote: > >> Thank you all.

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Liang-Chi Hsieh
Congrats! Welcome all! Dongjoon Hyun-2 wrote > Welcome everyone! :D > > Bests, > Dongjoon. > > On Tue, Jul 14, 2020 at 11:21 AM Xiao Li > lixiao@ > wrote: > >> Welcome, Dilip, Huaxin and Jungtaek! >> >> Xiao >> >> On Tue, Jul 14, 2020 at 11:02 AM Holden Karau > holden@ > >> wrote: >>

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Dongjoon Hyun
Welcome everyone! :D Bests, Dongjoon. On Tue, Jul 14, 2020 at 11:21 AM Xiao Li wrote: > Welcome, Dilip, Huaxin and Jungtaek! > > Xiao > > On Tue, Jul 14, 2020 at 11:02 AM Holden Karau > wrote: > >> So excited to have our committer pool growing with these awesome folks, >> welcome y'all! >> >>

Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Dongjoon Hyun
Hi, Yi. Could you explain why you think that is a blocker? For the given example from the JIRA description, spark.udf.register("key", udf((m: Map[String, String]) => m.keys.head.toInt)) Seq(Map("1" -> "one", "2" -> "two")).toDF("a").createOrReplaceTempView("t") checkAnswer(sql("SELECT key(a)

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Xiao Li
Welcome, Dilip, Huaxin and Jungtaek! Xiao On Tue, Jul 14, 2020 at 11:02 AM Holden Karau wrote: > So excited to have our committer pool growing with these awesome folks, > welcome y'all! > > On Tue, Jul 14, 2020 at 10:59 AM Driesprong, Fokko > wrote: > >> Welcome! >> >> Op di 14 jul. 2020 om

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Holden Karau
So excited to have our committer pool growing with these awesome folks, welcome y'all! On Tue, Jul 14, 2020 at 10:59 AM Driesprong, Fokko wrote: > Welcome! > > Op di 14 jul. 2020 om 19:53 schreef shane knapp ☠ : > >> welcome, all! >> >> On Tue, Jul 14, 2020 at 10:37 AM Matei Zaharia >> wrote:

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Driesprong, Fokko
Welcome! Op di 14 jul. 2020 om 19:53 schreef shane knapp ☠ : > welcome, all! > > On Tue, Jul 14, 2020 at 10:37 AM Matei Zaharia > wrote: > >> Hi all, >> >> The Spark PMC recently voted to add several new committers. Please join >> me in welcoming them to their new roles! The new committers are:

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread shane knapp ☠
welcome, all! On Tue, Jul 14, 2020 at 10:37 AM Matei Zaharia wrote: > Hi all, > > The Spark PMC recently voted to add several new committers. Please join me > in welcoming them to their new roles! The new committers are: > > - Huaxin Gao > - Jungtaek Lim > - Dilip Biswal > > All three of them

Re: Welcoming some new Apache Spark committers

2020-07-14 Thread Reynold Xin
Welcome all! On Tue, Jul 14, 2020 at 10:36 AM, Matei Zaharia < matei.zaha...@gmail.com > wrote: > > > > Hi all, > > > > The Spark PMC recently voted to add several new committers. Please join me > in welcoming them to their new roles! The new committers are: > > > > - Huaxin Gao > -

Welcoming some new Apache Spark committers

2020-07-14 Thread Matei Zaharia
Hi all, The Spark PMC recently voted to add several new committers. Please join me in welcoming them to their new roles! The new committers are: - Huaxin Gao - Jungtaek Lim - Dilip Biswal All three of them contributed to Spark 3.0 and we’re excited to have them join the project. Matei and

Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Yi Wu
This probably be a blocker: https://issues.apache.org/jira/browse/SPARK-32307 On Tue, Jul 14, 2020 at 11:13 PM Sean Owen wrote: > https://issues.apache.org/jira/browse/SPARK-32234 ? > > On Tue, Jul 14, 2020 at 9:57 AM Shivaram Venkataraman > wrote: > > > > Hi all > > > > Just wanted to check

Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Sean Owen
https://issues.apache.org/jira/browse/SPARK-32234 ? On Tue, Jul 14, 2020 at 9:57 AM Shivaram Venkataraman wrote: > > Hi all > > Just wanted to check if there are any blockers that we are still waiting for > to start the new release process. > > Thanks > Shivaram >

Re: [DISCUSS] Apache Spark 3.0.1 Release

2020-07-14 Thread Shivaram Venkataraman
Hi all Just wanted to check if there are any blockers that we are still waiting for to start the new release process. Thanks Shivaram On Sun, Jul 5, 2020, 06:51 wuyi wrote: > Ok, after having another look, I think it only affects local cluster deploy > mode, which is for testing only. > > >

Re: [PSA] Apache Spark uses GitHub Actions to run the tests

2020-07-14 Thread Hyukjin Kwon
Perfect. Plus, Github Actions is only for master branch at this moment. BTW, I think we can enable Java(Scala) doc build and dependency test back in Jenkins for simplicity. Seems like the Jenkins machine came back to normal. 2020년 7월 14일 (화) 오후 4:08, Wenchen Fan 님이 작성: > To clarify, we need to

Re: [PSA] Apache Spark uses GitHub Actions to run the tests

2020-07-14 Thread Wenchen Fan
To clarify, we need to wait for: 1. Java documentation build test in github actions 2. dependency test in github actions 3. either github action all green or jenkin pass If the PR touches Kinesis, or it uses other profiles, we must wait for jenkins to pass. Do I miss something? On Tue, Jul 14,

[PSA] Apache Spark uses GitHub Actions to run the tests

2020-07-14 Thread Hyukjin Kwon
Hi dev, Github Actions build was introduced to run the regular Spark test cases at https://github.com/apache/spark/pull/29057and https://github.com/apache/spark/pull/29086. This is virtually the duplication of default Jenkins PR builder at this moment. The only differences are: - Github Actions