Hi, I'm new to Livy and after a few days of digging through all the examples and possible combinations I've finally been able to achieve what I want - namely, I'm trying to create a large DataFrame and then call createOrReplaceTempView to keep this table in memory that can be re-used across sessions
My preferred approach would be to use the http batch api, but after googling and trialing, the spark session doesn't seem to be capable of being re-used and most notably I can't send the session id with the batch job POST across subsequent sessions. Is that correct? I also tried to see if it was possible to share sessions via the programmatic API as well but I didn't see a place where I can set the session id. I have got a shared spark dataframe working via http POST + sessions but this has its own drawbacks too. I have found that migrating working scala class code into the json body with string escaping quotes etc. very cumbersome and very trial and error. If my only choice is to use http sessions in order to re-use the DataFrame I was wondering if anyone can advise any best practices on how I might be able to re-use my existing + compiling scala code into the json code body of my http POST to tighten that trial and error approach? My preferred approach for sharing sessions would be: 1. batch api via http 2. programmatic api 3. sessions via http (only b/c there are no alternatives) Would appreciate if someone can advise Thx
