Are you pre-caching them in memory? On Apr 4, 2015 3:29 AM, "SamyaMaiti" <samya.maiti2...@gmail.com> wrote:
> Reduce *spark.sql.shuffle.partitions* from default of 200 to total number > of > cores. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/4-seconds-to-count-13M-lines-Does-it-make-sense-tp22360p22374.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
