What about using cache() or save as a global temp table for subsequent access?
Sent using Zoho Mail ============ Forwarded message ============ From : Affan
Syed <as...@an10.io> To : "spark users"<user@spark.apache.org> Date : Thu, 25
Oct 2018 10:58:43 +0330 Subject : Having access to spark results ============
Forwarded message ============ Spark users, We really would want to get an
input here about how the results from a Spark Query will be accessible to a
web-application. Given Spark is a well used in the industry I would have
thought that this part would have lots of answers/tutorials about it, but I
didnt find anything. Here are a few options that come to mind 1) Spark results
are saved in another DB ( perhaps a traditional one) and a request for query
returns the new table name for access through a paginated query. That seems
doable, although a bit convoluted as we need to handle the completion of the
query. 2) Spark results are pumped into a messaging queue from which a socket
server like connection is made. What confuses me is that other connectors to
spark, like those for Tableau, using something like JDBC should have all the
data (not the top 500 that we typically can get via Livy or other REST
interfaces to Spark). How do those connectors get all the data through a single
connection? Can someone with expertise help in bringing clarity. Thank you.
Affan ᐧ ᐧ
