Hi, Marco,
val data =
spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")
The data now include a feature column with name "features",
val featureIndexer = new VectorIndexer()
.setInputCol("features") <------ Here specify the "features"
column to index.
.setOutputCol("indexedFeatures")
Thanks.
On Sat, Dec 16, 2017 at 6:26 AM, Marco Mistroni <mmistr...@gmail.com> wrote:
> HI all
> i am trying to run a sample decision tree, following examples here (for
> Mllib)
>
> https://spark.apache.org/docs/latest/ml-classification-
> regression.html#decision-tree-classifier
>
> the example seems to use a Vectorindexer, however i am missing something.
> How does the featureIndexer knows which columns are features?
> Isnt' there something missing? or the featuresIndexer is able to figure
> out by itself
> which columns of teh DAtaFrame are features?
>
> val labelIndexer = new StringIndexer()
> .setInputCol("label")
> .setOutputCol("indexedLabel")
> .fit(data)// Automatically identify categorical features, and index
> them.val featureIndexer = new VectorIndexer()
> .setInputCol("features")
> .setOutputCol("indexedFeatures")
> .setMaxCategories(4) // features with > 4 distinct values are treated as
> continuous.
> .fit(data)
>
> Using this code i am getting back this exception
>
> Exception in thread "main" java.lang.IllegalArgumentException: Field
> "features" does not exist.
> at
> org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:266)
> at
> org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:266)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at org.apache.spark.sql.types.StructType.apply(StructType.scala:265)
> at
> org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:40)
> at
> org.apache.spark.ml.feature.VectorIndexer.transformSchema(VectorIndexer.scala:141)
> at
> org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
> at
> org.apache.spark.ml.feature.VectorIndexer.fit(VectorIndexer.scala:118)
>
> what am i missing?
>
> w/kindest regarsd
>
> marco
>
>
