Could you try different ranks and see whether the task size changes?
We do use YtY in the closure, which should work the same as broadcast.
If that is the case, it should be safe to ignore this warning.
-Xiangrui
On Thu, Apr 23, 2015 at 4:52 AM, Christian S. Perone
christian.per...@gmail.com
All these warnings come from ALS iterations, from flatMap and also from
aggregate, for instance the origin of the state where the flatMap is
showing these warnings (w/ Spark 1.3.0, they are also shown in Spark 1.3.1):
org.apache.spark.rdd.RDD.flatMap(RDD.scala:296)
This is the size of the serialized task closure. Is stage 246 part of
ALS iterations, or something before or after it? -Xiangrui
On Tue, Apr 21, 2015 at 10:36 AM, Christian S. Perone
christian.per...@gmail.com wrote:
Hi Sean, thanks for the answer. I tried to call repartition() on the input
I think maybe you need more partitions in your input, which might make
for smaller tasks?
On Tue, Apr 21, 2015 at 2:56 AM, Christian S. Perone
christian.per...@gmail.com wrote:
I keep seeing these warnings when using trainImplicit:
WARN TaskSetManager: Stage 246 contains a task of very large
I keep seeing these warnings when using trainImplicit:
WARN TaskSetManager: Stage 246 contains a task of very large size (208 KB).
The maximum recommended task size is 100 KB.
And then the task size starts to increase. Is this a known issue ?
Thanks !
--
Blog http://blog.christianperone.com |
the similarity between 2 users
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-tp22553.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-tp22553.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e
Is there any way that i can see the similarity table of 2 users in that
algorithm? by that i mean the similarity between 2 users
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-tp22553.html
Sent from the Apache Spark User List
Is there any way that i can see the similarity table of 2 users in that
algorithm?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-tp22552.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-tp22552.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
Spark version: 1.0.2
Number of workers: 9
core per worker: 16
memory per worker: 120GB
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-failed-to-run-with-rank-1000
.nabble.com/MLlib-Collaborative-Filtering-failed-to-run-with-rank-1000-tp15692.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-failed-to-run-with-rank-1000-tp15692p15697.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail
of hardware.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Collaborative-Filtering-failed-to-run-with-rank-1000-tp15692p15697.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
14 matches
Mail list logo
