Re: Python friendly API for Spark 3.0

2018-09-14 Thread Nicholas Chammas
Do we need to ditch Python 2 support to provide type hints? I don’t think so. Python lets you specify typing stubs that provide the same benefit without forcing Python 3. 2018년 9월 14일 (금) 오후 8:01, Holden Karau 님이 작성: > > > On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson wrote: > >> To be clear,

Re: Python friendly API for Spark 3.0

2018-09-14 Thread Holden Karau
On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson wrote: > To be clear, is this about "python-friendly API" or "friendly python API" ? > Well what would you consider to be different between those two statements? I think it would be good to be a bit more explicit, but I don't think we should

Re: Python friendly API for Spark 3.0

2018-09-14 Thread Erik Erlandson
To be clear, is this about "python-friendly API" or "friendly python API" ? On the python side, it might be nice to take advantage of static typing. Requires python 3.6 but with python 2 going EOL, a spark-3.0 might be a good opportunity to jump the python-3-only train. On Fri, Sep 14, 2018 at

Python friendly API for Spark 3.0

2018-09-14 Thread Holden Karau
Since we're talking about Spark 3.0 in the near future (and since some recent conversation on a proposed change reminded me) I wanted to open up the floor and see if folks have any ideas on how we could make a more Python friendly API for 3.0? I'm planning on taking some time to look at other

RE: [DISCUSS][CORE] Exposing application status metrics via a source

2018-09-14 Thread Luca Canali
Hi Stavros, All, Interesting topic, I add here some thoughts and personal opinions on it: I find too the metrics system quite useful for the use case of building Grafana dashboards as opposed to scraping logs and/or using the Event Listener infrastructure, as you mentioned in your mail. A few