[discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Reynold Xin
I took a look at the commit messages in git log -- it looks like the individual commit messages are not that useful to include, but do make the commit messages more verbose. They are usually just a bunch of extremely concise descriptions of bug fixes, merges, etc: cb3f12d [xxx] add whitespace

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Sean Owen
+1 to removing them. Sometimes there are 50+ commits because people have been merging from master into their branch rather than rebasing. On Sat, Jul 18, 2015 at 8:48 AM, Reynold Xin r...@databricks.com wrote: I took a look at the commit messages in git log -- it looks like the individual

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Ted Yu
+1 to removing commit messages. On Jul 18, 2015, at 1:35 AM, Sean Owen so...@cloudera.com wrote: +1 to removing them. Sometimes there are 50+ commits because people have been merging from master into their branch rather than rebasing. On Sat, Jul 18, 2015 at 8:48 AM, Reynold Xin

Writing to multiple outputs in Spark

2015-07-18 Thread Silas Davis
*tl;dr hadoop and cascading* *provide ways of writing tuples to multiple output files based on key, but the plain RDD interface doesn't seem to and it should.* I have been looking into ways to write to multiple outputs in Spark. It seems like a feature that is somewhat missing from Spark. The

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Ram Sriharsha
+1 Sent from my iPhone On Jul 18, 2015, at 2:44 PM, Patrick Wendell pwend...@gmail.com wrote: +1 from me too On Sat, Jul 18, 2015 at 3:32 AM, Ted Yu yuzhih...@gmail.com wrote: +1 to removing commit messages. On Jul 18, 2015, at 1:35 AM, Sean Owen so...@cloudera.com wrote: +1

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Patrick Wendell
+1 from me too On Sat, Jul 18, 2015 at 3:32 AM, Ted Yu yuzhih...@gmail.com wrote: +1 to removing commit messages. On Jul 18, 2015, at 1:35 AM, Sean Owen so...@cloudera.com wrote: +1 to removing them. Sometimes there are 50+ commits because people have been merging from master into their

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Reynold Xin
A single commit message consisting of: 1. Pull request title (which includes JIRA number and component, e.g. [SPARK-1234][MLlib]) 2. Pull request description 3. List of authors contributing to the patch The main thing that changes is 3: we used to also include the individual commits to the

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Mridul Muralidharan
Just to clarify, the proposal is to have a single commit msg giving the jira and pr id? That sounds like a good change to have. Regards Mridul On Saturday, July 18, 2015, Reynold Xin r...@databricks.com wrote: I took a look at the commit messages in git log -- it looks like the individual

Re: Expression.resolved unmatched with the correct values in catalyst?

2015-07-18 Thread Ted Yu
What if you move your addition to before line 64 (in master branch there is case for if e.checkInputDataTypes().isFailure): case c: Cast if !c.resolved = Cheers On Wed, Jul 15, 2015 at 12:47 AM, Takeshi Yamamuro linguin@gmail.com wrote: Hi, devs I found that the case of

If gmail, check sparm

2015-07-18 Thread Mridul Muralidharan
https://plus.google.com/+LinusTorvalds/posts/DiG9qANf5PA I have noticed a bunch of mails from dev@ and github going to spam - including spark maliing list. Might be a good idea for dev, committers to check if they are missing things in their spam folder if on gmail. Regards, Mridul

Dynamic resource allocation in Standalone mode

2015-07-18 Thread Dogtail Ray
Hi all, I am planning to dynamically increase or decrease the number of executors allocated to an application during runtime, and it is similar to dynamic resource allocation, which is only feasible in Spark on Yarn mode. Any suggestions on how to implement this feature in Standalone mode? My

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Mridul Muralidharan
Thanks for detailing, definitely sounds better. +1 Regards Mridul On Saturday, July 18, 2015, Reynold Xin r...@databricks.com wrote: A single commit message consisting of: 1. Pull request title (which includes JIRA number and component, e.g. [SPARK-1234][MLlib]) 2. Pull request

Re: If gmail, check sparm

2015-07-18 Thread Ted Yu
Interesting read. I did find a lot of Spark mails in Spam folder. Thanks Mridul On Jul 18, 2015, at 10:25 AM, Mridul Muralidharan mri...@gmail.com wrote: https://plus.google.com/+LinusTorvalds/posts/DiG9qANf5PA I have noticed a bunch of mails from dev@ and github going to spam -