Thank you Georg, Gene for your ideas. For now, I am using ”Futures” to asynchronously run a background thread that periodically creates a new dataframe fetching latest data from underlying table, and re-registers temp view with the same name as used by main thread’s static dataframe.
This looks to be working for me now, but if this solution leads to other problems, I will look for persisted views in hive / Alluxio. Regards, Hemanth From: Gene Pang <gene.p...@gmail.com> Date: Saturday, 22 April 2017 at 0.30 To: Georg Heiler <georg.kf.hei...@gmail.com> Cc: Hemanth Gudela <hemanth.gud...@qvantel.com>, Tathagata Das <tathagata.das1...@gmail.com>, "user@spark.apache.org" <user@spark.apache.org> Subject: Re: Spark structured streaming: Is it possible to periodically refresh static data frame? Hi Georg, Yes, that should be possible with Alluxio. Tachyon was renamed to Alluxio. This article on how Alluxio is used for a Spark streaming use case<https://www.alluxio.com/blog/qunar-performs-real-time-data-analytics-up-to-300x-faster-with-alluxio> may be helpful. Thanks, Gene On Fri, Apr 21, 2017 at 8:22 AM, Georg Heiler <georg.kf.hei...@gmail.com<mailto:georg.kf.hei...@gmail.com>> wrote: You could write your views to hive or maybe tachyon. Is the periodically updated data big? Hemanth Gudela <hemanth.gud...@qvantel.com<mailto:hemanth.gud...@qvantel.com>> schrieb am Fr. 21. Apr. 2017 um 16:55: Being new to spark, I think I need your suggestion again. #2 you can always define a batch Dataframe and register it as view, and then run a background then periodically creates a new Dataframe with updated data and re-registers it as a view with the same name I seem to have misunderstood your statement and tried registering static dataframe as a temp view (“myTempView”) using createOrReplaceView in one spark session, and tried re-registering another refreshed dataframe as temp view with same name (“myTempView”) in another session. However, with this approach, I have failed to achieve what I’m aiming for, because views are local to one spark session. From spark 2.1.0 onwards, Global view is a nice feature, but still would not solve my problem, because global view cannot be updated. So after much thinking, I understood that you would have meant to use a background running process in the same spark job that would periodically create a new dataframe and re-register temp view with same name, within the same spark session. Could you please give me some pointers to documentation on how to create such asynchronous background process in spark streaming? Is Scala’s “Futures” the way to achieve this? Thanks, Hemanth From: Tathagata Das <tathagata.das1...@gmail.com<mailto:tathagata.das1...@gmail.com>> Date: Friday, 21 April 2017 at 0.03 To: Hemanth Gudela <hemanth.gud...@qvantel.com<mailto:hemanth.gud...@qvantel.com>> Cc: Georg Heiler <georg.kf.hei...@gmail.com<mailto:georg.kf.hei...@gmail.com>>, "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Spark structured streaming: Is it possible to periodically refresh static data frame? Here are couple of ideas. 1. You can set up a Structured Streaming query to update in-memory table. Look at the memory sink in the programming guide - http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-sinks So you can query the latest table using a specified table name, and also join that table with another stream. However, note that this in-memory table is maintained in the driver, and so you have be careful about the size of the table. 2. If you cannot define a streaming query in the slow moving due to unavailability of connector for your streaming data source, then you can always define a batch Dataframe and register it as view, and then run a background then periodically creates a new Dataframe with updated data and re-registers it as a view with the same name. Any streaming query that joins a streaming dataframe with the view will automatically start using the most updated data as soon as the view is updated. Hope this helps. On Thu, Apr 20, 2017 at 1:30 PM, Hemanth Gudela <hemanth.gud...@qvantel.com<mailto:hemanth.gud...@qvantel.com>> wrote: Thanks Georg for your reply. But I’m not sure if I fully understood your answer. If you meant to join two streams (one reading Kafka, and another reading database table), then I think it’s not possible, because 1. According to documentation<http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#data-sources>, Structured streaming does not support database as a streaming source 2. Joining between two streams is not possible yet. Regards, Hemanth From: Georg Heiler <georg.kf.hei...@gmail.com<mailto:georg.kf.hei...@gmail.com>> Date: Thursday, 20 April 2017 at 23.11 To: Hemanth Gudela <hemanth.gud...@qvantel.com<mailto:hemanth.gud...@qvantel.com>>, "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Spark structured streaming: Is it possible to periodically refresh static data frame? What about treating the static data as a (slow) stream as well? Hemanth Gudela <hemanth.gud...@qvantel.com<mailto:hemanth.gud...@qvantel.com>> schrieb am Do., 20. Apr. 2017 um 22:09 Uhr: Hello, I am working on a use case where there is a need to join streaming data frame with a static data frame. The streaming data frame continuously gets data from Kafka topics, whereas static data frame fetches data from a database table. However, as the underlying database table is getting updated often, I must somehow manage to refresh my static data frame periodically to get the latest information from underlying database table. My questions: 1. Is it possible to periodically refresh static data frame? 2. If refreshing static data frame is not possible, is there a mechanism to automatically stop & restarting spark structured streaming job, so that every time the job restarts, the static data frame gets updated with latest information from underlying database table. 3. If 1) and 2) are not possible, please suggest alternatives to achieve my requirement described above. Thanks, Hemanth