Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-22 Thread Hemanth Gudela
<hemanth.gud...@qvantel.com>, Tathagata Das <tathagata.das1...@gmail.com>, "user@spark.apache.org" <user@spark.apache.org> Subject: Re: Spark structured streaming: Is it possible to periodically refresh static data frame? Hi Georg, Yes, that should be possible with

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-21 Thread Gene Pang
>> >> >> Thanks, >> >> Hemanth >> >> >> >> >> >> *From: *Tathagata Das <tathagata.das1...@gmail.com> >> >> >> *Date: *Friday, 21 April 2017 at 0.03 >> *To: *Hemanth Gudela <hemanth.gud...@qvantel.com> >>

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-21 Thread Georg Heiler
t; <http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#data-sources>, > Structured streaming does not support database as a streaming source > > 2. Joining between two streams is not possible yet. > > > > Regards, > > Hemanth

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-21 Thread Hemanth Gudela
:georg.kf.hei...@gmail.com>> Date: Thursday, 20 April 2017 at 23.11 To: Hemanth Gudela <hemanth.gud...@qvantel.com<mailto:hemanth.gud...@qvantel.com>>, "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-20 Thread Georg Heiler
y, 21 April 2017 at 0.03 > *To: *Hemanth Gudela <hemanth.gud...@qvantel.com> > *Cc: *Georg Heiler <georg.kf.hei...@gmail.com>, "user@spark.apache.org" < > user@spark.apache.org> > > > *Subject: *Re: Spark structured streaming: Is it possible to perio

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-20 Thread Hemanth Gudela
<tathagata.das1...@gmail.com> Date: Friday, 21 April 2017 at 0.03 To: Hemanth Gudela <hemanth.gud...@qvantel.com> Cc: Georg Heiler <georg.kf.hei...@gmail.com>, "user@spark.apache.org" <user@spark.apache.org> Subject: Re: Spark structured streaming: Is it possible to perio

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-20 Thread Tathagata Das
at 23.11 > *To: *Hemanth Gudela <hemanth.gud...@qvantel.com>, "user@spark.apache.org" > <user@spark.apache.org> > *Subject: *Re: Spark structured streaming: Is it possible to periodically > refresh static data frame? > > > > What about treating the static data as a (s

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-20 Thread Hemanth Gudela
manth Gudela <hemanth.gud...@qvantel.com>, "user@spark.apache.org" <user@spark.apache.org> Subject: Re: Spark structured streaming: Is it possible to periodically refresh static data frame? What about treating the static data as a (slow) stream as well? Hemanth

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-20 Thread Georg Heiler
What about treating the static data as a (slow) stream as well? Hemanth Gudela schrieb am Do., 20. Apr. 2017 um 22:09 Uhr: > Hello, > > > > I am working on a use case where there is a need to join streaming data > frame with a static data frame. > > The streaming

Spark structured streaming: Is it possible to periodically refresh static data frame?

2017-04-20 Thread Hemanth Gudela
Hello, I am working on a use case where there is a need to join streaming data frame with a static data frame. The streaming data frame continuously gets data from Kafka topics, whereas static data frame fetches data from a database table. However, as the underlying database table is getting