[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[yoelb]

Github user yoelb commented on the issue:

https://github.com/apache/spark/pull/19687
  
I have the same issue @ManchesterUnited16  , did you find a solution?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
@ManchesterUnited16 Since I tested your codes and it works for me, I'm 
going to merge this PR and backport it to 2.2. If it's a real issue, feel free 
to open a ticket with steps to reproduce it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
> can you show me you maven dependency when you ran the program,thank you 
very much!

maven dependency is used only when I compiled the codes. I used the 
official Spark 2.2.0 to compile the codes, and built Spark based on this PR 
(commit hash: f88fb50aec853f45dfc213cbeb9851a270ba3cbe) and ran my problem 
using it. Could you show how did you compile the codes, how did you build 
Spark, and how did you run the codes?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[ManchesterUnited16]

Github user ManchesterUnited16 commented on the issue:

https://github.com/apache/spark/pull/19687
  
can you show me you maven dependency when you ran the program,thank you 
very much!






At 2017-11-09 13:37:46, "Shixiong Zhu"  wrote:


@ManchesterUnited16 I ran your codes and didn't see 
NotSerializableException. How did you patch Spark with my PR?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
@ManchesterUnited16 I ran your codes and didn't see 
`NotSerializableException`. How did you patch Spark with my PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[ManchesterUnited16]

Github user ManchesterUnited16 commented on the issue:

https://github.com/apache/spark/pull/19687
  
import org.apache.spark.ml.evaluation.RegressionEvaluator
import org.apache.spark.ml.recommendation.{ALS, ALSModel}
import org.apache.spark.ml.recommendation.ALS.Rating
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, Row}
// $example off$
import org.apache.spark.sql.SparkSession

object ALSExample {
// $example on$
case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
def parseRating(str: String): Rating = {
val fields = str.split("::")
assert(fields.size == 4)
Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat,fields(3).toLong)
}
// $example off$

def main(args: Array[String]) {
val spark = SparkSession
.builder
.master("local")
.appName("ALSExample")
.getOrCreate()
import spark.implicits._

// $example on$
val ratings: DataFrame = 
spark.read.textFile("D:\\xcar\\Spark_MLib\\ml_2.11.1\\src\\data\\mllib\\als\\sample_movielens_ratings.txt")
.map(parseRating)
.toDF()
val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2))

// Build the recommendation model using ALS on the training data
val als = new ALS()
.setMaxIter(5)
.setRegParam(0.01)
.setUserCol("userId")
.setItemCol("movieId")
.setRatingCol("rating")
val model: ALSModel = als.fit(training)

// Evaluate the model by computing the RMSE on the test data
// Note we set cold start strategy to 'drop' to ensure we don't get 
NaN evaluation metrics
model.setColdStartStrategy("drop")
val predictions: DataFrame = model.transform(test)

val evaluator: RegressionEvaluator = new RegressionEvaluator()
.setMetricName("rmse")
.setLabelCol("rating")
.setPredictionCol("prediction")
val rmse = evaluator.evaluate(predictions)
println(s"Root-mean-square error = $rmse")

// Generate top 10 movie recommendations for each user
val userRecs: DataFrame = model.recommendForAllUsers(10)
// Generate top 10 user recommendations for each movie
val movieRecs: DataFrame = model.recommendForAllItems(10)

// Generate top 10 movie recommendations for a specified set of users
//val users = ratings.select(als.getUserCol).distinct().limit(3)
//val userSubsetRecs = model.recommendForUserSubset(users, 10)
//// Generate top 10 user recommendations for a specified set of 
movies
//val movies = ratings.select(als.getItemCol).distinct().limit(3)
//val movieSubSetRecs = model.recommendForItemSubset(movies, 10)
// $example off$
userRecs.show()

//movieRecs.show()
//userSubsetRecs.show()
//movieSubSetRecs.show()

spark.stop()
}

}
this is my code ,and when I run the line"userRecs.show()",there are some 
error "java.io.NotSerializableException: 
scala.reflect.api.TypeTags$PredefTypeCreator"






At 2017-11-09 05:29:55, "Shixiong Zhu"  wrote:


@zsxwing commented on this pull request.

In 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala:

> @@ -441,4 +443,28 @@ class ExpressionEncoderSuite extends PlanTest with 
AnalysisTest {
   }
 }
   }
+
+  /**
+   * Verify the size of scala.reflect.runtime.JavaUniverse.undoLog before 
and after `func` to
+   * ensure we don't leak Scala reflection garbage.
+   *
+   * @see 
org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects
+   */
+  private def verifyNotLeakingReflectionObjects[T](func: => T): T = {
+def undoLogSize: Int = {
+  import scala.reflect.runtime.{JavaUniverse, universe}


No special reason. I changed to use the fully qualified class name now.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19687
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83608/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19687
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19687
  
**[Test build #83608 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83608/testReport)**
 for PR 19687 at commit 
[`f88fb50`](https://github.com/apache/spark/commit/f88fb50aec853f45dfc213cbeb9851a270ba3cbe).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
> LGTM, is it targeted for branch 2.2 too?

Yeah, I will backport this to 2.2 once this PR gets merged.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
@ManchesterUnited16 `cleanUpReflectionObjects` introduces new closures but 
those are created in `object ScalaReflection` so should be safe. Did you 
backport my PR to an old Spark version?
 If so, you probably hit https://github.com/apache/spark/pull/17639.

Anyway, I added some defensive codes in the test to make sure encoders are 
serializable.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19687
  
**[Test build #83608 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83608/testReport)**
 for PR 19687 at commit 
[`f88fb50`](https://github.com/apache/spark/commit/f88fb50aec853f45dfc213cbeb9851a270ba3cbe).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
> java.io.NotSerializableException: 
scala.reflect.api.TypeTags$PredefTypeCreator

@ManchesterUnited16 Probably this change may capture some unnecessary 
objects. Could you provide your codes to help me investigate it?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19687
  
LGTM, is it targeted for branch 2.2 too?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[ManchesterUnited16]

Github user ManchesterUnited16 commented on the issue:

https://github.com/apache/spark/pull/19687
  
java.io.NotSerializableException: 
scala.reflect.api.TypeTags$PredefTypeCreator


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[ManchesterUnited16]

Github user ManchesterUnited16 commented on the issue:

https://github.com/apache/spark/pull/19687
  

there is something wrong in my program


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19687
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83571/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19687
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19687
  
**[Test build #83571 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83571/testReport)**
 for PR 19687 at commit 
[`c03811f`](https://github.com/apache/spark/commit/c03811ff006058987fa8d5fb9f7d097b9acc9ac5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19687
  
**[Test build #83571 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83571/testReport)**
 for PR 19687 at commit 
[`c03811f`](https://github.com/apache/spark/commit/c03811ff006058987fa8d5fb9f7d097b9acc9ac5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/19687
  
cc @cloud-fan 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org