Hello,
I've asked the following question [1] on Stackoverflow but didn't get an
answer, yet. I use now this channel to give it more visibility, and
hopefully find someone who can help.
"*Context.* I have tens of SQL queries stored in separate files. For
benchmarking purposes, I created an application that iterates through
each of those query files and passes it to a standalone Spark
application. This latter /first/ parses the query, extracts the used
tables, registers them (using: registerTempTable() in Spark < 2 and
createOrReplaceTempView() in Spark 2), and executes effectively the
query (spark.sql()).
*Challenge.* Since registering the tables can sometimes be time
consuming, I would like to register the tables only once when they are
first used, and keep that in form of metadata that can readily be used
in the subsequent queries without the need to re-register the tables
again. It's a sort of intra-job caching but not any of the caching Spark
offers (table caching), as far as I know.
Is that possible? if not can anyone suggest another approach to
accomplish the same goal (i.e., iterating through separate query files
and run a querying Spark application without registering the tables that
have already been registered before)."
[1]:
http://stackoverflow.com/questions/40549924/sparksql-intra-sparksql-application-table-registration
Cheers,
Mohamed
