I have a set of transformers (each with specific parameters) in spark 1.3.1. I have two versions, one that works and one that does not:
1.- working version //featureprovidertransformer contains already a set of ml params class DemographicTransformer(override val uid: String) extends FeatureProviderTransformer { def this() = this(Identifiable.randomUID("demo-transformer")) override def copy(extra: ParamMap): DemographicTransformer = defaultCopy(extra) .... } 2.- not working version class DemographicTransformer(override val uid: String) extends FeatureProviderTransformer { def this() = this(Identifiable.randomUID("demo-transformer")) override def copy(extra: ParamMap): DemographicTransformer = defaultCopy(extra) *//add another transformer parameter* final val anotherParam: Param[String] = new Param[String](this, "anotherParam", "dummy parameter") .... } Somehow adding an *anotherParam* to my class make it fail, with the following error: [info] java.lang.NullPointerException: [info] at org.apache.spark.ml.param.Params$$anonfun$hasParam$1.apply(params.scala:408) [info] at org.apache.spark.ml.param.Params$$anonfun$hasParam$1.apply(params.scala:408) [info] at scala.collection.IndexedSeqOptimized$$anonfun$exists$1.apply(IndexedSeqOptimized.scala:40) [info] at scala.collection.IndexedSeqOptimized$$anonfun$exists$1.apply(IndexedSeqOptimized.scala:40) [info] at scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:189) [info] at scala.collection.mutable.ArrayOps$ofRef.segmentLength(ArrayOps.scala:108) [info] at scala.collection.GenSeqLike$class.prefixLength(GenSeqLike.scala:92) [info] at scala.collection.mutable.ArrayOps$ofRef.prefixLength(ArrayOps.scala:108) [info] at scala.collection.IndexedSeqOptimized$class.exists(IndexedSeqOptimized.scala:40) [info] at scala.collection.mutable.ArrayOps$ofRef.exists(ArrayOps.scala:108) Debugging the params.scala class shows me that actually adding *anotherParam* *replace all parameters by a single one called allParams.* *Does anyone have any idea of what I may be doing wrong. My guess is that I am doing something weird in my class hierarchy but can not figure out what.* Thanks! -- Cesar Flores