ASF board report for May

2019-05-06 Thread Matei Zaharia
It’s time to submit Spark's quarterly ASF board report on May 15th, so I wanted to run the report by everyone to make sure we’re not missing something. Let me know whether I missed anything: Apache Spark is a fast and general engine for large-scale data processing. It offers

Re: [VOTE] Release Apache Spark 2.4.3

2019-05-06 Thread Xiao Li
This vote passes! I'll follow up with a formal release announcement soon. +1: Michael Heuer (non-binding) Gengliang Wang (non-binding) Sean Owen (binding) Felix Cheung (binding) Wenchen Fan (binding) Herman van Hovell (binding) Xiao Li (binding) Cheers, Xiao antonkulaga 于2019年5月6日周一 下午2:36写道:

Re: [VOTE] Release Apache Spark 2.4.3

2019-05-06 Thread antonkulaga
>Hadoop 3 has not been supported in 2.4.x. 2.12 has been since 2.4.0, I see. I thought it was as I saw many posts about configuring Spark for Hadoop 3 as well as hadoop 3 based spark docker containers -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

[METRICS] Metrics names inconsistent between executions

2019-05-06 Thread Anton Kirillov
Hi everyone! We are currently working on building a unified monitoring/alerting solution for Spark and would like to rely on Spark's own metrics to avoid divergence from the upstream. One of the challenges is to support metrics coming from multiple Spark applications running on a cluster:

DataSourceV2 community sync notes - 1 May 2019

2019-05-06 Thread Ryan Blue
Here are my notes for the latest DSv2 community sync. As usual, if you have comments or corrections, please reply. If you’d like to be invited to the next sync, email me directly. Everyone is welcome to attend. *Attendees*: Ryan Blue John Zhuge Andrew Long Bruce Robbins Dilip Biswal Gengliang

Re: [VOTE] Release Apache Spark 2.4.3

2019-05-06 Thread Wenchen Fan
+1. The Scala version problem has been resolved, which is the main motivation of 2.4.3. On Mon, May 6, 2019 at 12:38 AM Felix Cheung wrote: > I ran basic tests on R, r-hub etc. LGTM. > > +1 (limited - I didn’t get to run other usual tests) > > -- > *From:* Sean Owen