[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707421#comment-14707421 ] Apache Spark commented on SPARK-8400: - User 'BryanCutler' has created a pull request for this issue: https://github.com/apache/spark/pull/8363 ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Assignee: Bryan Cutler Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707409#comment-14707409 ] Bryan Cutler commented on SPARK-8400: - No problem! It does LocalIndexEncoder once training is started and after some data operations, so you are right that it is probably better to make this check earlier. I'll submit a PR on 1.3 for this. ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Assignee: Bryan Cutler Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706182#comment-14706182 ] Xiangrui Meng commented on SPARK-8400: -- Sorry for my late reply! We check numBlocks in LocalIndexEncoder. However, I'm not sure whether this happens before any data shuffling. It might be better to check numUserBlocks and numItemBlocks directly. ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Assignee: Bryan Cutler Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[
https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625483#comment-14625483
]
Bryan Cutler commented on SPARK-8400:
-
Hi [~mengxr], just in case you missed my comment in the PR about adding an
error message in branch 1.3 for this:
{quote}
{noformat}
using branch-1.3 I gave ALS a -1 block size and got the following exception:
[info] ALSSuite:
[info] - more blocks than ratings *** FAILED *** (1 second, 112 milliseconds)
[info] java.lang.IllegalArgumentException: requirement failed: numBlocks must
be positive but found -1.
[info] at scala.Predef$.require(Predef.scala:233)
[info] at
org.apache.spark.ml.recommendation.ALS$LocalIndexEncoder.(ALS.scala:1164)
Seems like it's already fixed to me, what do you think?
{noformat}
{quote}
ml.ALS doesn't handle -1 block size
---
Key: SPARK-8400
URL: https://issues.apache.org/jira/browse/SPARK-8400
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 1.3.1
Reporter: Xiangrui Meng
Assignee: Bryan Cutler
Under spark.mllib, if number blocks is set to -1, we set the block size
automatically based on the input partition size. However, this behavior is
not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not
work, but no error messages will show.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621365#comment-14621365 ] Apache Spark commented on SPARK-8400: - User 'BryanCutler' has created a pull request for this issue: https://github.com/apache/spark/pull/7333 ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617292#comment-14617292 ] Xiangrui Meng commented on SPARK-8400: -- [~bryanc] Are you still working on this issue? ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617359#comment-14617359 ] Bryan Cutler commented on SPARK-8400: - Hi [~mengxr], yes I am. I'll hopefully have a PR soon for you to check out. ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592593#comment-14592593 ] Bryan Cutler commented on SPARK-8400: - Ok, sounds good. ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8400) ml.ALS doesn't handle -1 block size
[ https://issues.apache.org/jira/browse/SPARK-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589039#comment-14589039 ] Bryan Cutler commented on SPARK-8400: - I could do this. Just to clarify, if the user sets -1 should it print an error and not continue or handle it automatically as in spark.mllib? ml.ALS doesn't handle -1 block size --- Key: SPARK-8400 URL: https://issues.apache.org/jira/browse/SPARK-8400 Project: Spark Issue Type: Bug Components: ML Affects Versions: 1.3.1 Reporter: Xiangrui Meng Under spark.mllib, if number blocks is set to -1, we set the block size automatically based on the input partition size. However, this behavior is not preserved in the spark.ml API. If user sets -1 in Spark 1.3, it will not work, but no error messages will show. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
