I don't think there's anything specific to CDH that you need to know, other than it ought to set things up sanely for you.
Sandy did a couple posts about tuning: http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/ http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ I don't think there's such a thing as one optimal configuration. It depends very heavily on your workload. First you need to have a look at your app, really. All the tuning in the world isn't going to make an unnecessary shuffle as fast as eliminating it. On Thu, Apr 16, 2015 at 6:02 PM, Manish Gupta 8 <mgupt...@sapient.com> wrote: > Hi, > > > > Is there a document/link that describes the general configuration settings > to achieve maximum Spark Performance while running on CDH5? In our > environment, we did lot of changes (and still doing it) to get decent > performance otherwise our 6 node dev cluster with default configurations, > lags behind a single laptop running Spark. > > > > Having a standard checklist (taking a base node size of 4-CPU, 16GB RAM) > would be really great. Any pointers in this regards will be really helpful. > > > > We are running Spark 1.2.0 on CDH 5.3.0. > > > > Thanks, > > > > Manish Gupta > > Specialist | Sapient Global Markets > > > > Green Boulevard (Tower C) > > 3rd & 4th Floor > > Plot No. B-9A, Sector 62 > > Noida 201 301 > > Uttar Pradesh, India > > > > Tel: +91 (120) 479 5000 > > Fax: +91 (120) 479 5001 > > Email: mgupt...@sapient.com > > > > sapientglobalmarkets.com > > > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > (your) computer. > > > > ***Please consider the environment before printing this email.*** > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
