[DISCUSS][CORE] Exposing application status metrics via a source

Hi all, I have a PR https://github.com/apache/spark/pull/22381 that exposes application status metrics (related jira: SPARK-25394). So far metrics tooling needs to scrape the metrics rest api to get metrics like job delay, stages failed, stages completed etc. >From devops perspective it is good

Re: RE: [VOTE] SPARK 2.3.2 (RC5)

2.3.2 includes crucial changes for Python 3.7 support as well. Thanks, Jon On 2018/09/07 03:26:30, Sharanabasappa G Keriwaddi wrote: > Hi – > > Are there any blocking issues open for 2.3.2? > > 2.3.1 had few critical issues, I feel it would be better to publish 2.3.2 > with all those

Kubernetes Big-Data-SIG notes, September 12

Spark Pod template parameters is mostly done. The main remaining design discussion is around how (or whether) to specify which container on the pod template is the driver pod. HDFS We had some discussions about the possibility of adding an HDFS

Re: python tests: any reason for a huge tests.py?

So I've had some offline discussion around this, so I'd like to clarify. SPARK-25344 maybe some non-trivial work to do, as its significant refactoring. But can we agree on an *immediate* first step: all new python tests should go into their own files? is there some reason to not do that right