@spark.apache.org
Subject: Re: Gradient Descent with large model size
Yes, remember that your bandwidth is the maximum number of bytes per second
that can be shipped to the driver. So if you've got 5 blocks that size, then it
looks like you're basically saturating the network.
Aggregation trees help for many
@gmail.com]
> Sent: Saturday, October 17, 2015 2:24 PM
> To: Joseph Bradley
> Cc: Ulanov, Alexander; dev@spark.apache.org
> Subject: Re: Gradient Descent with large model size
>
> Yes, remember that your bandwidth is the maximum number of bytes per second
> that can be shipped
For those numbers of partitions, I don't think you'll actually use tree
aggregation. The number of partitions needs to be over a certain threshold
(>= 7) before treeAggregate really operates on a tree structure:
Bradley [mailto:jos...@databricks.com]
Sent: Wednesday, October 14, 2015 11:35 PM
To: Ulanov, Alexander
Cc: dev@spark.apache.org
Subject: Re: Gradient Descent with large model size
For those numbers of partitions, I don't think you'll actually use tree
aggregation. The number of partitions needs
