Are you pre-caching them in memory?
On Apr 4, 2015 3:29 AM, "SamyaMaiti" wrote:
> Reduce *spark.sql.shuffle.partitions* from default of 200 to total number
> of
> cores.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/4-seconds-to-count-13M-lines-Does-it-make-sense-tp22360p22374.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
