: MapValues and Shuffle Reads
Hi Darin,
When you say you see 400GB of shuffle writes from the first code
snippet, what do you mean? There is no action in that first set, so it
won't do anything. By itself, it won't do any shuffle writing, or anything
else for that matter.
Most likely
Hi Darin,
When you say you see 400GB of shuffle writes from the first code snippet,
what do you mean? There is no action in that first set, so it won't do
anything. By itself, it won't do any shuffle writing, or anything else for
that matter.
Most likely, the .count() on your second code
really want to do in the first place.
Thanks again for your insights.
Darin.
From: Imran Rashid iras...@cloudera.com
To: Darin McBeath ddmcbe...@yahoo.com
Cc: User user@spark.apache.org
Sent: Tuesday, February 17, 2015 3:29 PM
Subject: Re: MapValues and Shuffle
