[GitHub] [spark] srowen commented on a change in pull request #26469: [SPARK-29844] Improper unpersist strategy in ml.recommendation.ASL.train

2019-11-11 Thread GitBox
srowen commented on a change in pull request #26469: [SPARK-29844] Improper 
unpersist strategy in ml.recommendation.ASL.train
URL: https://github.com/apache/spark/pull/26469#discussion_r344747970
 
 

 ##
 File path: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
 ##
 @@ -1041,13 +1041,13 @@ object ALS extends DefaultParamsReadable[ALS] with 
Logging {
   .persist(finalRDDStorageLevel)
 if (finalRDDStorageLevel != StorageLevel.NONE) {
   userIdAndFactors.count()
-  itemIdAndFactors.count()
-  itemFactors.unpersist()
   userInBlocks.unpersist()
   userOutBlocks.unpersist()
-  itemInBlocks.unpersist()
   itemOutBlocks.unpersist()
   blockRatings.unpersist()
+  itemIdAndFactors.count()
 
 Review comment:
   Yes, but itemFactors depends on the previous user RDDs, including 
userOutBlocks. I think this is done on purpose. EDIT: Hm OK but it's persisted, 
so shouldn't matter as long as itemFactors is unpersisted afterwards. This is a 
bit of an optimization right? just tries to free up more space before 
persisting the item stuff.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen commented on a change in pull request #26469: [SPARK-29844] Improper unpersist strategy in ml.recommendation.ASL.train

2019-11-11 Thread GitBox
srowen commented on a change in pull request #26469: [SPARK-29844] Improper 
unpersist strategy in ml.recommendation.ASL.train
URL: https://github.com/apache/spark/pull/26469#discussion_r344747970
 
 

 ##
 File path: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
 ##
 @@ -1041,13 +1041,13 @@ object ALS extends DefaultParamsReadable[ALS] with 
Logging {
   .persist(finalRDDStorageLevel)
 if (finalRDDStorageLevel != StorageLevel.NONE) {
   userIdAndFactors.count()
-  itemIdAndFactors.count()
-  itemFactors.unpersist()
   userInBlocks.unpersist()
   userOutBlocks.unpersist()
-  itemInBlocks.unpersist()
   itemOutBlocks.unpersist()
   blockRatings.unpersist()
+  itemIdAndFactors.count()
 
 Review comment:
   Yes, but itemFactors depends on the previous user RDDs, including 
userOutBlocks. I think this is done on purpose.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] srowen commented on a change in pull request #26469: [SPARK-29844] Improper unpersist strategy in ml.recommendation.ASL.train

2019-11-11 Thread GitBox
srowen commented on a change in pull request #26469: [SPARK-29844] Improper 
unpersist strategy in ml.recommendation.ASL.train
URL: https://github.com/apache/spark/pull/26469#discussion_r344731778
 
 

 ##
 File path: mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala
 ##
 @@ -1041,13 +1041,13 @@ object ALS extends DefaultParamsReadable[ALS] with 
Logging {
   .persist(finalRDDStorageLevel)
 if (finalRDDStorageLevel != StorageLevel.NONE) {
   userIdAndFactors.count()
-  itemIdAndFactors.count()
-  itemFactors.unpersist()
   userInBlocks.unpersist()
   userOutBlocks.unpersist()
-  itemInBlocks.unpersist()
   itemOutBlocks.unpersist()
   blockRatings.unpersist()
+  itemIdAndFactors.count()
 
 Review comment:
   I don't see the problem here immediately. Doesn't this depend on some of the 
cached user RDDs?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org