Re: [DISCUSS] New sections in Github Pull Request description template

2019-07-23 Thread Reynold Xin
I like the spirit, but not sure about the exact proposal. Take a look at k8s': https://raw.githubusercontent.com/kubernetes/kubernetes/master/.github/PULL_REQUEST_TEMPLATE.md On Tue, Jul 23, 2019 at 8:27 PM, Hyukjin Kwon wrote: > (Plus, it helps to track history too. Spark's commit logs are

Re: [DISCUSS] New sections in Github Pull Request description template

2019-07-23 Thread Hyukjin Kwon
(Plus, it helps to track history too. Spark's commit logs are growing and now it's pretty difficult to track the history and see what change introduced a specific behaviour) 2019년 7월 24일 (수) 오후 12:20, Hyukjin Kwon 님이 작성: > Hi all, > > I would like to discuss about some new sections under "##

[DISCUSS] New sections in Github Pull Request description template

2019-07-23 Thread Hyukjin Kwon
Hi all, I would like to discuss about some new sections under "## What changes were proposed in this pull request?": ### Do the changes affect _any_ user/dev-facing input or output? (Please answer yes or no. If yes, answer the questions below) ### What was the previous behavior? (Please

Re: DataSourceV2 sync notes - 10 July 2019

2019-07-23 Thread Ryan Blue
I agree that the long-term solution is much farther away, but I'm not sure it is a good idea to do this in the optimizer. Maybe we could find a good way to do it, but the initial complication required before we moved to push-down to the conversion to physical plan was really bad. Plus, this has

Re: DataSourceV2 sync notes - 10 July 2019

2019-07-23 Thread Wenchen Fan
Hi Ryan, Thanks for summarizing and sending out the meeting notes! Unfortunately, I missed the last sync, but the topics are really interesting, especially the stats integration. The ideal solution I can think of is to refactor the optimizer/planner and move all the stats-based optimization to