Could you try different ranks and see whether the task size changes?
We do use YtY in the closure, which should work the same as broadcast.
If that is the case, it should be safe to ignore this warning.
-Xiangrui
On Thu, Apr 23, 2015 at 4:52 AM, Christian S. Perone
christian.per...@gmail.com
All these warnings come from ALS iterations, from flatMap and also from
aggregate, for instance the origin of the state where the flatMap is
showing these warnings (w/ Spark 1.3.0, they are also shown in Spark 1.3.1):
org.apache.spark.rdd.RDD.flatMap(RDD.scala:296)
This is the size of the serialized task closure. Is stage 246 part of
ALS iterations, or something before or after it? -Xiangrui
On Tue, Apr 21, 2015 at 10:36 AM, Christian S. Perone
christian.per...@gmail.com wrote:
Hi Sean, thanks for the answer. I tried to call repartition() on the input
I think maybe you need more partitions in your input, which might make
for smaller tasks?
On Tue, Apr 21, 2015 at 2:56 AM, Christian S. Perone
christian.per...@gmail.com wrote:
I keep seeing these warnings when using trainImplicit:
WARN TaskSetManager: Stage 246 contains a task of very large
I keep seeing these warnings when using trainImplicit:
WARN TaskSetManager: Stage 246 contains a task of very large size (208 KB).
The maximum recommended task size is 100 KB.
And then the task size starts to increase. Is this a known issue ?
Thanks !
--
Blog http://blog.christianperone.com |
