Anthony Truchet created SPARK-18471:
---------------------------------------
Summary: In treeAggregate, generate (big) zeros instead of sending
them.
Key: SPARK-18471
URL: https://issues.apache.org/jira/browse/SPARK-18471
Project: Spark
Issue Type: Improvement
Components: MLlib, Spark Core
Reporter: Anthony Truchet
Priority: Minor
When using optimization routine like LBFGS, treeAggregate curently sends the
zero vector as part of the closure. This zero can be huge (e.g. ML vectors with
millions of zeros) but can be easily generated.
Several option are possible (upcoming patches to come soon for some of them).
On is to provide a treeAggregateWithZeroGenerator method (either in core on in
MLlib) which wrap treeAggregate in an option and generate the zero if None.
Another one is to rewrite treeAggregate to wrap an underlying implementation
which use a zero generator directly.
There might be other better alternative we have not spotted...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
