[ https://issues.apache.org/jira/browse/SPARK-8708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608194#comment-14608194 ]
Sean Owen commented on SPARK-8708: ---------------------------------- Does that actually make 5 partitions? I see that's what's requested, but are the items evenly distributed? The computation doesn't use 1 partition, so the question is why the result would have 1 partition. It might if you have a small number of products that all get into one partition for whatever reason, I think, since the final join is on product. I have one more idea on the PR ... > MatrixFactorizationModel.predictAll() populates single partition only > --------------------------------------------------------------------- > > Key: SPARK-8708 > URL: https://issues.apache.org/jira/browse/SPARK-8708 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.3.0 > Reporter: Antony Mayi > > When using mllib.recommendation.ALS the RDD returned by .predictAll() has all > values pushed into single partition despite using quite high parallelism. > This degrades performance of further processing (I can obviously run > .partitionBy()) to balance it but that's still too costly (ie if running > .predictAll() in loop for thousands of products) and should be possible to do > it rather somehow on the model (automatically)). > Bellow is an example on tiny sample (same on large dataset): > {code:title=pyspark} > >>> r1 = (1, 1, 1.0) > >>> r2 = (1, 2, 2.0) > >>> r3 = (2, 1, 2.0) > >>> r4 = (2, 2, 2.0) > >>> r5 = (3, 1, 1.0) > >>> ratings = sc.parallelize([r1, r2, r3, r4, r5], 5) > >>> ratings.getNumPartitions() > 5 > >>> users = ratings.map(itemgetter(0)).distinct() > >>> model = ALS.trainImplicit(ratings, 1, seed=10) > >>> predictions_for_2 = model.predictAll(users.map(lambda u: (u, 2))) > >>> predictions_for_2.glom().map(len).collect() > [0, 0, 3, 0, 0] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org