Re: Spark 1.3.1 on Yarn not using all given capacity

2015-10-06 Thread Cesar Berezowski
3 cores* not 8 César. > Le 6 oct. 2015 à 19:08, Cesar Berezowski a écrit : > > I deployed hdp 2.3.1 and got spark 1.3.1, spark 1.4 is supposed to be > available as technical preview I think > > vendor’s forum ? you mean hortonworks' ? > > -- > Update on m

Job on Yarn not using all given capacity ends up failing

2015-10-05 Thread Cesar Berezowski
Hi, I recently upgraded from 1.2.1 to 1.3.1 (through HDP). I have a job that does a cartesian product on two datasets (2K and 500K lines minimum) to do string matching. I updated it to use Dataframes because the old code wouldn’t run anymore (deprecated RDD functions). It used to run very w