Can you try the following: - Use Kryo Serializer - Enable RDD Compression - Repartition the data (Use hash partition, then all the similar keys will go in the same partition)
Thanks Best Regards On Thu, Jan 22, 2015 at 4:05 AM, vantoniuk <[email protected]> wrote: > I have temporary fix for my case. My sample file was 2G / 50M lines in > size. > My initial configuration was 1000 splits. > > Based on my understanding of distributed algorithms, number of splits can > affect the memory pressure in operations such as distinct and reduceByKey. > So i tried to reduce the number of splits from 1000 to 100. Now I can run > distinct and reduceByKey on files that are 2G / 50M lines. > > Unfortunately it still doesn't scale well. > > Thanks. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Exception-in-connection-from-worker-to-worker-tp20983p21302.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
