Re: Ask for ARM CI for spark

2019-07-09 Thread Tianhua huang
Hi all, I am glad to tell you there is a new progress of build/test spark on aarch64 server, the tests are running, see the build/test detail log https://logs.openlabtesting.org/logs/1/1/419fcb11764048d5a3cda186ea76dd43249e1f97/check/spark-build-arm64/75cc6f5/job-output.txt.gz and the aarch64

Re: Release Apache Spark 2.4.4 before 3.0.0

2019-07-09 Thread Dongjoon Hyun
Thank you for the reply, Sean. Sure. 2.4.x should be a LTS version. The main reason of 2.4.4 release (before 3.0.0) is to have a better basis for comparison to 3.0.0. For example, SPARK-27798 had an old bug, but its correctness issue is only exposed at Spark 2.4.3. It would be great if we can

Re: Release Apache Spark 2.4.4 before 3.0.0

2019-07-09 Thread Sean Owen
We will certainly want a 2.4.4 release eventually. In fact I'd expect 2.4.x gets maintained for longer than the usual 18 months, as it's the last 2.x branch. It doesn't need to happen before 3.0, but could. Usually maintenance releases happen 3-4 months apart and the last one was 2 months ago. If

Release Apache Spark 2.4.4 before 3.0.0

2019-07-09 Thread Dongjoon Hyun
Hi, All. Spark 2.4.3 was released two months ago (8th May). As of today (9th July), there exist 45 fixes in `branch-2.4` including the following correctness or blocker issues. - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for decimals not fitting in long - SPARK-26045

Re: Contribution help needed for sub-tasks of an umbrella JIRA - port *.sql tests to improve coverage of Python, Pandas, Scala UDF cases

2019-07-09 Thread Hyukjin Kwon
It's alright - thanks for that. Anyone can take a look. This is an open source project :D. 2019년 7월 9일 (화) 오후 8:18, Stavros Kontopoulos < stavros.kontopou...@lightbend.com>님이 작성: > I can try one and see how it goes, although not familiar with the area. > > Stavros > > On Tue, Jul 9, 2019 at 6:17

Re: Contribution help needed for sub-tasks of an umbrella JIRA - port *.sql tests to improve coverage of Python, Pandas, Scala UDF cases

2019-07-09 Thread Stavros Kontopoulos
I can try one and see how it goes, although not familiar with the area. Stavros On Tue, Jul 9, 2019 at 6:17 AM Hyukjin Kwon wrote: > Hi all, > > I am currently targeting to improve Python, Pandas UDFs Scala UDF test > cases by integrating our existing *.sql files at >

Re: Opinions wanted: how much to match PostgreSQL semantics?

2019-07-09 Thread Dongjoon Hyun
Thank you, Sean and all. One decision was made swiftly today. I believe that we can move forward case-by-case for the others until the feature freeze (3.0 branch cut). Bests, Dongjoon. On Mon, Jul 8, 2019 at 13:03 Marco Gaido wrote: > Hi Sean, > > Thanks for bringing this up. Honestly, my