I'm running into an error that's not making a lot of sense to me, and couldn't find sufficient info on the web to answer it myself. BTW, you can reply at Stack Overflow too: http://stackoverflow.com/questions/36254005/nosuchelementexception-in-chisqselector-fit-method-version-1-6-0
I've written code to generate a list of (String, ArrayBuffer[String]) pairs and then use HashingTF to convert the features column to vectors (bc it's for NLP research on parsing where I end up with a whole lot of unique features; long story). Then I convert the string labels using StringIndexer. I get the "key not found" error when running ChiSqSelector.fit on the training data. The stack trace points to a hashmap lookup in ChiSqTest for labels. This struck me as strange, because I could sort of reason that perhaps I was using it wrong and had not somehow accounted for unseen labels -- except this was the fit method on training data. Anyway, here's the interesting bit of my code followed by the important part of the stack trace. Any help would be very much appreciated!! val parSdp = sc.parallelize(sdp.take(10)) // it dies on a small amount of data val insts: RDD[(String, ArrayBuffer[String])] = parSdp.flatMap(x=> TrainTest.transformGraphSpark(x)) val indexer = new StringIndexer() .setInputCol("labels") .setOutputCol("labelIndex") val instDF = sqlContext.createDataFrame(insts) .toDF("labels","feats") val hash = new HashingTF() .setInputCol("feats") .setOutputCol("hashedFeats") .setNumFeatures(1000000) val readyDF = hash.transform(indexer .fit(instDF) .transform(instDF)) val selector = new ChiSqSelector() .setNumTopFeatures(100) .setFeaturesCol("hashedFeats") .setLabelCol("labelIndex") .setOutputCol("selectedFeatures") val Array(training, dev,test) = readyDF.randomSplit(Array(0.8,0.1,0.1), seed = 12345) val chisq = selector.fit(training) And the stack trace: java.util.NoSuchElementException: key not found: 23.0 at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:58) at scala.collection.MapLike$class.apply(MapLike.scala:141) at scala.collection.AbstractMap.apply(Map.scala:58) at org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4$$anonfun$apply$4.apply(ChiSqTest.scala:131) at org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4$$anonfun$apply$4.apply(ChiSqTest.scala:129) at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153) at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306) at org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:129) at org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:125) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredFeatures(ChiSqTest.scala:125) at org.apache.spark.mllib.stat.Statistics$.chiSqTest(Statistics.scala:176) at org.apache.spark.mllib.feature.ChiSqSelector.fit(ChiSqSelector.scala:193) at org.apache.spark.ml.feature.ChiSqSelector.fit(ChiSqSelector.scala:86) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:89) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:122) ... etc etc I also realized that by changing the size of sdp.take larger (to 100) above I get a different error: java.lang.IllegalArgumentException: Chi-squared statistic undefined for input matrix due to0 sum in column [4]. at org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredMatrix(ChiSqTest.scala:229) at org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:134) at org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:125) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredFeatures(ChiSqTest.scala:125) at org.apache.spark.mllib.stat.Statistics$.chiSqTest(Statistics.scala:176) at org.apache.spark.mllib.feature.ChiSqSelector.fit(ChiSqSelector.scala:193) at org.apache.spark.ml.feature.ChiSqSelector.fit(ChiSqSelector.scala:86) at $iwC$$iwC.<init>(<console>:96) at $iwC.<init>(<console>:130) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchElementException-in-ChiSqSelector-fit-method-version-1-6-0-tp26614.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org