What about treating the static data as a (slow) stream as well? Hemanth Gudela <hemanth.gud...@qvantel.com> schrieb am Do., 20. Apr. 2017 um 22:09 Uhr:
> Hello, > > > > I am working on a use case where there is a need to join streaming data > frame with a static data frame. > > The streaming data frame continuously gets data from Kafka topics, whereas > static data frame fetches data from a database table. > > > > However, as the underlying database table is getting updated often, I must > somehow manage to refresh my static data frame periodically to get the > latest information from underlying database table. > > > > My questions: > > 1. Is it possible to periodically refresh static data frame? > > 2. If refreshing static data frame is not possible, is there a > mechanism to automatically stop & restarting spark structured streaming > job, so that every time the job restarts, the static data frame gets > updated with latest information from underlying database table. > > 3. If 1) and 2) are not possible, please suggest alternatives to > achieve my requirement described above. > > > > Thanks, > > Hemanth >
