Re: Not per-key state in spark streaming

Thank you very much for your reply , Daniel On Thu, Dec 8, 2016 at 7:07 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > There's no need to extend Spark's API, look at mapWithState for examples. > > On Thu, Dec 8, 2016 at 4:49 AM, Anty Rao wrote: > >> >> >> On

Re: Not per-key state in spark streaming

There's no need to extend Spark's API, look at mapWithState for examples. On Thu, Dec 8, 2016 at 4:49 AM, Anty Rao wrote: > > > On Wed, Dec 7, 2016 at 7:42 PM, Anty Rao wrote: > >> Hi >> I'm new to Spark. I'm doing some research to see if spark streaming

Re: Not per-key state in spark streaming

On Wed, Dec 7, 2016 at 7:42 PM, Anty Rao wrote: > Hi > I'm new to Spark. I'm doing some research to see if spark streaming can > solve my problem. I don't want to keep per-key state,b/c my data set is > very huge and keep a little longer time, it not viable to keep all per key

Re: Not per-key state in spark streaming

@Daniel Thanks for your reply. I will try it. On Wed, Dec 7, 2016 at 8:47 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi Anty, > What you could do is keep in the state only the existence of a key and > when necessary pull it from a secondary state store like HDFS or HBASE. > >

Re: Not per-key state in spark streaming

Hi Anty, What you could do is keep in the state only the existence of a key and when necessary pull it from a secondary state store like HDFS or HBASE. Daniel On Wed, Dec 7, 2016 at 1:42 PM, Anty Rao wrote: > Hi > I'm new to Spark. I'm doing some research to see if spark

Not per-key state in spark streaming

Hi I'm new to Spark. I'm doing some research to see if spark streaming can solve my problem. I don't want to keep per-key state,b/c my data set is very huge and keep a little longer time, it not viable to keep all per key state in memory.Instead, i want to have a bloom filter based state. Does it