On Thu, Oct 13, 2016 at 1:42 PM, drewrobb <drewr...@gmail.com> wrote:
> When using spark on mesos and deploying a job in cluster mode using > dispatcher, there appears to be no memory overhead configuration for the > launched driver processes ("--driver-memory" is the same as Xmx which is > the > same as the memory quota). This makes it almost a guarantee that a long > running driver will be OOM killed by mesos. Yarn cluster mode has an > equivalent option -- spark.yarn.driver.memoryOverhead. Is there some way > to > configure driver memory overhead that I'm missing? > > Bigger picture question-- Is it even best practice to deploy long running > spark streaming jobs using dispatcher? I could alternatively launch the > driver by itself using marathon for example, where it would be trivial to > grant the process additional memory. > > Thanks! > > I wrote a simple wrapper to provide a little head room, I'm currently using the Chronos framework to run my spark jobs which provides some handy environment variables like CHRONOS_RESOURCE_MEM which is what's offered from Mesos. execString.append('--driver-memory {:.0f}M'.format(.7 * float(os.environ.get('CHRONOS_RESOURCE_MEM',params.get('memory'))))) execString.append('--executor-memory {:.0f}M'.format(.98 * float(os.environ.get('CHRONOS_RESOURCE_MEM',params.get('memory'))))) execString.append('--jars {} {} {}'.format(params.get('classpath'), params.get('jarfile'), params.get('optargs'))) i.e. SPARK_MESOS_DRIVER_MEM = CHRONOS_RESOURCE_MEM * .7 We basically set the upper limit for the driver to use 70% of what's allocated in total to Mesos. This will help get around Mesos killing long running jobs because of OOM. -- [image: Orchard Platform] <http://www.orchardplatform.com/> *Rodrick Brown */ *SRE - DevOPs* 9174456839 / rodr...@orchardplatform.com Orchard Platform 101 5th Avenue, 4th Floor, New York, NY -- *NOTICE TO RECIPIENTS*: This communication is confidential and intended for the use of the addressee only. If you are not an intended recipient of this communication, please delete it immediately and notify the sender by return email. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. This communication does not constitute an offer to sell or a solicitation of an indication of interest to purchase any loan, security or any other financial product or instrument, nor is it an offer to sell or a solicitation of an indication of interest to purchase any products or services to any persons who are prohibited from receiving such information under applicable law. The contents of this communication may not be accurate or complete and are subject to change without notice. As such, Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes no representation regarding the accuracy or completeness of the information contained herein. The intended recipient is advised to consult its own professional advisors, including those specializing in legal, tax and accounting matters. Orchard does not provide legal, tax or accounting advice.