[GitHub] spark pull request #19661: [SPARK-22450][Core][Mllib]safely register class f...

[ConeyLiu]

Github user ConeyLiu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19661#discussion_r149553694
  
    --- Diff: 
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
    @@ -178,10 +178,40 @@ class KryoSerializer(conf: SparkConf)
         
kryo.register(Utils.classForName("scala.collection.immutable.Map$EmptyMap$"))
         kryo.register(classOf[ArrayBuffer[Any]])
     
    +    // We can't load those class directly in order to avoid unnecessary 
jar dependencies.
    +    // We load them safely, ignore it if the class not found.
    +    Seq("org.apache.spark.mllib.linalg.Vector",
    +      "org.apache.spark.mllib.linalg.DenseVector",
    +      "org.apache.spark.mllib.linalg.SparseVector",
    +      "org.apache.spark.mllib.linalg.Matrix",
    +      "org.apache.spark.mllib.linalg.DenseMatrix",
    +      "org.apache.spark.mllib.linalg.SparseMatrix",
    +      "org.apache.spark.ml.linalg.Vector",
    +      "org.apache.spark.ml.linalg.DenseVector",
    +      "org.apache.spark.ml.linalg.SparseVector",
    +      "org.apache.spark.ml.linalg.Matrix",
    +      "org.apache.spark.ml.linalg.DenseMatrix",
    +      "org.apache.spark.ml.linalg.SparseMatrix",
    +      "org.apache.spark.ml.feature.Instance",
    +      "org.apache.spark.ml.feature.OffsetInstance"
    +    ).flatMap(safeClassLoader(_)).foreach(kryo.register(_))
    --- End diff --
    
    Hi @cloud-fan , I tried the following code：
    ```scala
    flatMap(cn => 
Try{Utils.classForName(cn)}.toOption).foreach(kryo.register(_))
    ```
    and 
    ```scala
    flatMap{ cn =>
          try {
            val clazz = Utils.classForName(cn)
            Some(clazz)
          } catch {
            case _: ClassNotFoundException => None
          }
        }.foreach(kryo.register(_))
    ```
    
    Both reported the same errors:
    ```
    Error:(198, 18) type mismatch;
     found   : String => Iterable[Class[_$2]]( forSome { type _$2 })
     required: String => scala.collection.GenTraversableOnce[B]
        ).flatMap{cn => 
Option(Utils.classForName(cn))}.foreach(kryo.register(_))
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org