Writing to Parquet and querying the result via SparkSQL works great (except for some strange SQL parser errors). However the problem remains, how do I get that data back to a dashboard. So I guess I’ll have to use a database after all.
You can batch up data & store into parquet partitions as well. & query it using another SparkSQL shell, JDBC driver in SparkSQL is part 1.1 i believe.
