What version are you running? I think "spark.shuffle.use.netty" was a valid
option only in Spark 1.1, where the Netty stuff was strictly experimental.
Spark 1.2 contains an officially supported and much more thoroughly tested
version under the property "spark.shuffle.blockTransferService", which is
set to netty by default.
On Tue, Jan 13, 2015 at 9:26 PM, lihu <lihu...@gmail.com> wrote:
> Hi,
> I just test groupByKey method on a 100GB data, the cluster is 20
> machine, each with 125GB RAM.
>
> At first I set conf.set("spark.shuffle.use.netty", "false") and run
> the experiment, and then I set conf.set("spark.shuffle.use.netty", "true")
> again to re-run the experiment, but at the latter case, the GC time is much
> higher。
>
>
> I thought the latter one should be better, but it is not. So when should
> we use netty for network shuffle fetching?
>
>
>
