Thanks to all for the quick replies, they helped a lot. To answer a few of the follow-up questions ...
> 1. How did you fix this performance which I gather programmatically The main problem in my original code was that the <first time through the data frame> logic was not being executed when it should have been. This 'if' clause has the effect of creating a new data frame directly from the contents of the Cassandra data source, throwing away the old RDD lineage of the existing data frame. By incorrectly executing the 'else' clause instead, the new data frame was being created from the existing data frame again and again, extending the entire lineage with each iteration and degrading query times. Fixing the code stopped the lineage from constantly growing with each refresh. > 2. In your code have you set spark.conf.set("spark.sql.adaptive.enabled", > "true") Yes, this is already set. > 3. Depending on the size of your data both source and new data, do you > have any indication that your data in global temporary view is totally > cached. This should show in the storage tab in UI. If you have data on the > disk for then this will affect the performance Yes, the cached temporary view is in the storage tab with the expected amount of memory usage, although it is difficult to check the specific contents of this memory cache. > You could try checkpointing occasionally and see if that helps This was very helpful, as I was not familiar with the Spark checkpointing feature. In our application, we don't need fault-tolerant dataframes/RDDs, so I inserted localCheckpoint() calls in the code prior to the uncaching and re-caching steps. This seems to have made all queries run faster, with some improved immediately by a factor of 3! Based on these tests, it is clear that RDD lineage can have a major effect on the performance of Spark applications, whether or not the data has been cached in memory. Thanks again for the good advice! -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org