Hi, We need to query deeply nested Json structure. However query is on a single field at a nested level such as mean, median, mode.
I am aware of the sql explode function. df = df_nested.withColumn('exploded', explode(top)) But this is too slow. Is there any other strategy that could give us the best performance in querying nested json in Spark Dataset. Thanks