What's the type of the key? If the hash of key is different across slaves, then you could get this confusing results. We had met this similar results in Python, because of hash of None is different across machines.
Davies On Mon, Sep 8, 2014 at 8:16 AM, redocpot <[email protected]> wrote: > Update: > > Just test with HashPartitioner(8) and count on each partition: > > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657591*), (*6,658327*), (*7,658434*)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657594)*, (6,658326), (*7,658434*)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657592)*, (6,658326), (*7,658435*)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657591)*, (6,658326), (7,658434)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657592)*, (6,658326), (7,658435)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657592)*, (6,658326), (7,658435)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657592)*, (6,658326), (7,658435)), > List((0,657824), (1,658549), (2,659199), (3,658684), (4,659394), > *(5,657591)*, (6,658326), (7,658435)) > > The result is not identical for each execution. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/groupBy-gives-non-deterministic-results-tp13698p13702.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
