NoSuchElementException in ChiSqSelector fit method (version 1.6.0)
I'm running into an error that's not making a lot of sense to me, and
couldn't find sufficient info on the web to answer it myself. BTW, you can
reply at Stack Overflow too:
http://stackoverflow.com/questions/36254005/nosuchelementexception-in-chisqselector-fit-method-version-1-6-0
I've written code to generate a list of (String, ArrayBuffer[String]) pairs
and then use HashingTF to convert the features column to vectors (bc it's
for NLP research on parsing where I end up with a whole lot of unique
features; long story). Then I convert the string labels using StringIndexer.
I get the "key not found" error when running ChiSqSelector.fit on the
training data. The stack trace points to a hashmap lookup in ChiSqTest for
labels. This struck me as strange, because I could sort of reason that
perhaps I was using it wrong and had not somehow accounted for unseen labels
-- except this was the fit method on training data.
Anyway, here's the interesting bit of my code followed by the important part
of the stack trace. Any help would be very much appreciated!!
val parSdp = sc.parallelize(sdp.take(10)) // it dies on a small amount
of data
val insts: RDD[(String, ArrayBuffer[String])] =
parSdp.flatMap(x=> TrainTest.transformGraphSpark(x))
val indexer = new StringIndexer()
.setInputCol("labels")
.setOutputCol("labelIndex")
val instDF = sqlContext.createDataFrame(insts)
.toDF("labels","feats")
val hash = new HashingTF()
.setInputCol("feats")
.setOutputCol("hashedFeats")
.setNumFeatures(100)
val readyDF = hash.transform(indexer
.fit(instDF)
.transform(instDF))
val selector = new ChiSqSelector()
.setNumTopFeatures(100)
.setFeaturesCol("hashedFeats")
.setLabelCol("labelIndex")
.setOutputCol("selectedFeatures")
val Array(training, dev,test) = readyDF.randomSplit(Array(0.8,0.1,0.1),
seed = 12345)
val chisq = selector.fit(training)
And the stack trace:
java.util.NoSuchElementException: key not found: 23.0
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:58)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:58)
at
org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4$$anonfun$apply$4.apply(ChiSqTest.scala:131)
at
org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4$$anonfun$apply$4.apply(ChiSqTest.scala:129)
at
scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
at
scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at
org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:129)
at
org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:125)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at
org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredFeatures(ChiSqTest.scala:125)
at
org.apache.spark.mllib.stat.Statistics$.chiSqTest(Statistics.scala:176)
at
org.apache.spark.mllib.feature.ChiSqSelector.fit(ChiSqSelector.scala:193)
at
org.apache.spark.ml.feature.ChiSqSelector.fit(ChiSqSelector.scala:86)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:89)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:122)
... etc etc
I also realized that by changing the size of sdp.take larger (to 100) above
I get a different error:
java.lang.IllegalArgumentException: Chi-squared statistic undefined for
input matrix due to0 sum in column [4].
at
org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredMatrix(ChiSqTest.scala:229)
at
org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:134)
at
org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:125)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala.
Fwd: NoSuchElementException in ChiSqSelector fit method (version 1.6.0)
Hi All,
I'm running into an error that's not making a lot of sense to me, and
couldn't find sufficient info on the web to answer it myself.
BTW, you can also reply on Stack Overflow:
http://stackoverflow.com/questions/36254005/nosuchelementexception-in-chisqselector-fit-method-version-1-6-0
I've written code to generate a list of (String, ArrayBuffer[String]) pairs
and then use HashingTF to convert the features column to vectors (bc it's
for NLP research on parsing where I end up with a whole lot of unique
features; long story). Then I convert the string labels using
StringIndexer. I get the "key not found" error when running
ChiSqSelector.fit on the training data. The stack trace points to a hashmap
lookup in ChiSqTest for labels. This struck me as strange, because I could
sort of reason that perhaps I was using it wrong and had not somehow
accounted for unseen labels -- except this was the fit method on training
data.
Anyway, here's the interesting bit of my code followed by the important
part of the stack trace. Any help would be very much appreciated!!
> val parSdp = sc.parallelize(sdp.take(100)) // it dies on a small
> amount of data
> val insts: RDD[(String, ArrayBuffer[String])] =
> parSdp.flatMap(x=> TrainTest.transformGraphSpark(x))
>
> val indexer = new StringIndexer()
> .setInputCol("labels")
> .setOutputCol("labelIndex")
>
> val instDF = sqlContext.createDataFrame(insts)
> .toDF("labels","feats")
> val hash = new HashingTF()
> .setInputCol("feats")
> .setOutputCol("hashedFeats")
> .setNumFeatures(100)
> val readyDF = hash.transform(indexer
> .fit(instDF)
> .transform(instDF))
>
> val selector = new ChiSqSelector()
> .setNumTopFeatures(100)
> .setFeaturesCol("hashedFeats")
> .setLabelCol("labelIndex")
> .setOutputCol("selectedFeatures")
>
> val Array(training, dev,test) =
> readyDF.randomSplit(Array(0.9,0.1,0.1), seed = 12345)
>
> val chisq = selector.fit(training)
And the stack trace:
java.util.NoSuchElementException: key not found: 23.0
>
> at scala.collection.MapLike$class.default(MapLike.scala:228)
> at scala.collection.AbstractMap.default(Map.scala:58)
> at scala.collection.MapLike$class.apply(MapLike.scala:141)
> at scala.collection.AbstractMap.apply(Map.scala:58)
> at
> org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4$$anonfun$apply$4.apply(ChiSqTest.scala:131)
> at
> org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4$$anonfun$apply$4.apply(ChiSqTest.scala:129)
> at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
> at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
> at
> org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:129)
> at
> org.apache.spark.mllib.stat.test.ChiSqTest$$anonfun$chiSquaredFeatures$4.apply(ChiSqTest.scala:125)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at
> org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredFeatures(ChiSqTest.scala:125)
> at
> org.apache.spark.mllib.stat.Statistics$.chiSqTest(Statistics.scala:176)
> at
> org.apache.spark.mllib.feature.ChiSqSelector.fit(ChiSqSelector.scala:193)
> at
> org.apache.spark.ml.feature.ChiSqSelector.fit(ChiSqSelector.scala:86)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:89)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:122)
> ... etc etc
--
Best Wishes,
Joshua Cason
