Hi all,
is there a way to create a Spark SQL Row schema based on Scala data
types without creating a manual mapping?
That's the only example I can find which doesn't require
spark.sql.types.DataType already as input, but it requires to define
them as Strings.
* val struct = (new StructType) * .add("a", "int") * .add("b", "long") *
.add("c", "string")
Specifically I have an RDD where each element is a Map of 100s of
variables with different data types which I want to transform to a DataFrame
where the keys should end up as the column names:
Map ("Amean" -> 20.3, "Asize" -> 12, "Bmean" -> ....)
Is there a different possibility than building a mapping from the
values' .getClass to the Spark SQL DataTypes?
Thanks,
Fabian
