Re: spark code formatter?

2014-01-09 Thread Patrick Wendell
I'm also very wary of using a code formatter for the reasons already mentioned by Reynold. Does scaliform have a mode where it just provides style checks rather than reformat the code? This is something we really need for, e.g., reviewing the many submissions to the project. - Patrick On Wed,

Re: spark code formatter?

2014-01-09 Thread DB Tsai
Initially, we also had the same concern, so we started from limited set of rules. Gradually, we found that it increases the productivity and readability of our codebase. PS, Scalariform is compatible with the Scala Style Guide in the sense that, given the right preference settings, source code

yarn, fat-jars and lib_managed

2014-01-09 Thread Alex Cozzi
I am just starting out playing with spark on our hadoop 2.2 cluster and I have a question. The current way to submit jobs to the cluster is to create fat-jars with sbt assembly. This approach works but I think is less than optimal in many large hadoop installation: the way we interact with

RE: yarn, fat-jars and lib_managed

2014-01-09 Thread Liu, Raymond
I think you could put the spark jar and other jar your app depends on while not changes a lot on HDFS, and use --files or --addjars ( depends on the mode you run YarnClient/YarnStandalone ) to refer to them. And then just need to redeploy your thin app jar on each invoke. Best Regards, Raymond

Contribute SimRank algorightm to mllib

2014-01-09 Thread Shao, Saisai
Hi All, We would like to contribute SimRank algorithm to mllib. SimRank algorithm used to calculate similarity rank between two objects based on graph structure, details can be seen in (http://ilpubs.stanford.edu:8090/508/1/2001-41.pdf), here we implemented a matrix multiplication method based

Re: multinomial logistic regression

2014-01-09 Thread Michael Kun Yang
I just sent the pr, fixed a typo in the comment. Add some comments and unit test. Please let me know if you receive the patch. On Mon, Jan 6, 2014 at 9:18 PM, Michael Kun Yang kuny...@stanford.eduwrote: I will follow up the newtown one later On Mon, Jan 6, 2014 at 9:14 PM, Michael Kun Yang