Yes, just as my last post, using [] to access array data and "." to access
nested fields seems not work.
BTW, i have deeped into the code of the current master branch.
spark / sql / catalyst / src / main / scala / org / apache / spark / sql /
catalyst / plans / logical / LogicalPlan.scala
from line 72 (in the function def resolve(name: String):
Option[NamedExpression])
...
options.distinct match {
case (a, Nil) :: Nil => Some(a) // One match, no nested fields, use
it.
// One match, but we also need to extract the requested nested field.
case (a, nestedFields) :: Nil =>
a.dataType match {
case StructType(fields) =>
Some(Alias(nestedFields.foldLeft(a: Expression)(GetField),
nestedFields.last)())
case _ => None // Don't know how to resolve these field references
}
case Nil => None // No matches.
case ambiguousReferences =>
throw new TreeNodeException(
this, s"Ambiguous references to $name:
${ambiguousReferences.mkString(",")}")
}
...
seems only StructType is handled and the ArrayType is not.
So it seems to be bug or some feature not finished yet?
thx!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Query-the-nested-JSON-data-With-Spark-SQL-1-0-1-tp9544p9733.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
