Re: spark on yarn wastes one box (or 1 GB on each box) for am container

2016-02-09 Thread praveen S
How about running in client mode, so that the client from which it is run becomes the driver. Regards, Praveen On 9 Feb 2016 16:59, "Steve Loughran" wrote: > > > On 9 Feb 2016, at 06:53, Sean Owen wrote: > > > > > > I think you can let YARN over-commit RAM though, and allocate more > > memory t

Usage of SparkContext within a Web container

2016-01-13 Thread praveen S
Is use of SparkContext from a Web container a right way to process spark jobs or should we use spark-submit in a processbuilder? Are there any pros or cons of using SparkContext from a Web container..? How does zeppelin trigger spark jobs from the Web context?