Re: Notes on writing complex spark applications

2014-11-24 Thread Evan R. Sparks
Thanks Patrick, You raise a good point - for this to be useful it's imperative that it is updated with new versions of spark. My thought with putting it on the wiki was that it's lower friction for community members to edit, but it likely won't have the same level of quality control as the

Notes on writing complex spark applications

2014-11-23 Thread Evan R. Sparks
Hi all, Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been working on a short document about writing high performance Spark applications based on our experience developing MLlib, GraphX, ml-matrix, pipelines, etc. It may be a useful document both for users and new Spark

Re: Notes on writing complex spark applications

2014-11-23 Thread andy petrella
Cool! On Sun Nov 23 2014 at 5:58:03 PM Evan R. Sparks evan.spa...@gmail.com wrote: Hi all, Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been working on a short document about writing high performance Spark applications based on our experience developing MLlib, GraphX,

Re: Notes on writing complex spark applications

2014-11-23 Thread Sam Bessalah
Thanks Evan, this is great. On Nov 23, 2014 5:58 PM, Evan R. Sparks evan.spa...@gmail.com wrote: Hi all, Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been working on a short document about writing high performance Spark applications based on our experience developing

Re: Notes on writing complex spark applications

2014-11-23 Thread Inkyu Lee
Very helpful!! thank you very much! 2014-11-24 2:17 GMT+09:00 Sam Bessalah samkiller@gmail.com: Thanks Evan, this is great. On Nov 23, 2014 5:58 PM, Evan R. Sparks evan.spa...@gmail.com wrote: Hi all, Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been working

Re: Notes on writing complex spark applications

2014-11-23 Thread Patrick Wendell
Hey Evan, It might be nice to merge this into existing documentation. In particular, a lot of this could serve to update the current tuning section and programming guides. It could also work to paste this wholesale as a reference for Spark users, but in that case it's less likely to get updated