Shuffle produces one huge partition

this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-produces-one-huge-partition-tp23358.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscri

Re: Shuffle produces one huge partition

simpler way to solve this? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-produces-one-huge-partition-tp23358.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > -

Re: Shuffle produces one huge partition and many tiny partitions

at are known to appear very often are assigned random partitions instead of using the existing partitioning mechanism. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-produces-one-huge-partition-and-many-tiny-partitions-tp23358p23387.html Sent from t

Re: Shuffle produces one huge partition and many tiny partitions

-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-produces-one-huge-partition-and-many-tiny-partitions-tp23358p23387.html Sent from the Apache Spark User List mailing list archive at Nabble.com. -

Re: Shuffle produces one huge partition and many tiny partitions

eats certain "exception" keys differently. > These keys that are known to appear very often are assigned random > partitions instead of using the existing partitioning mechanism. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3

Re: Shuffle produces one huge partition and many tiny partitions

itioner that extends HashPartitioner.  It treats certain "exception" keys differently. These keys that are known to appear very often are assigned random partitions instead of using the existing partitioning mechanism. -- View this message in context: http://apache-spark-user-list.1001560.n3.n

Re: Shuffle produces one huge partition and many tiny partitions

n "exception" keys differently. > These keys that are known to appear very often are assigned random > partitions instead of using the existing partitioning mechanism. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Shuffl