Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

2019-09-06 Thread Shyam P
Difficult things in spark is debugging and tuning.

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

2019-09-06 Thread David Zhou
Not yet. Learning spark On Fri, Sep 6, 2019 at 2:17 PM Shyam P wrote: > cool ,but did you find a way or anyhelp or clue ? > > On Fri, Sep 6, 2019 at 11:40 PM David Zhou wrote: > >> I have the same question with yours >> >> On Thu, Sep 5, 2019 at 9:18 PM Shyam P wrote: >> >>> Hi, >>> >>> I am

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

2019-09-06 Thread Shyam P
cool ,but did you find a way or anyhelp or clue ? On Fri, Sep 6, 2019 at 11:40 PM David Zhou wrote: > I have the same question with yours > > On Thu, Sep 5, 2019 at 9:18 PM Shyam P wrote: > >> Hi, >> >> I am using spark-sql-2.4.1v to streaming in my PoC. >> >> how to refresh the loaded

Re: how to refresh the loaded non-streaming dataframe for each steaming batch ?

2019-09-06 Thread David Zhou
I have the same question with yours On Thu, Sep 5, 2019 at 9:18 PM Shyam P wrote: > Hi, > > I am using spark-sql-2.4.1v to streaming in my PoC. > > how to refresh the loaded dataframe from hdfs/cassandra table every time > new batch of stream processed ? What is the practice followed in general

how to refresh the loaded non-streaming dataframe for each steaming batch ?

2019-09-05 Thread Shyam P
Hi, I am using spark-sql-2.4.1v to streaming in my PoC. how to refresh the loaded dataframe from hdfs/cassandra table every time new batch of stream processed ? What is the practice followed in general to handle this kind of scenario? Below is the SOF link for more details .