Hi, We need to query deeply nested Json structure. However query is on a single field at a nested level such as mean, median, mode.
I am aware of the sql explode function.
df = df_nested.withColumn('exploded', explode(top))
But this is too slow.
Is there any other strategy that could give us the best performance in
querying nested json in Spark Dataset.
Thanks
