Hi Anty, What you could do is keep in the state only the existence of a key and when necessary pull it from a secondary state store like HDFS or HBASE.
Daniel On Wed, Dec 7, 2016 at 1:42 PM, Anty Rao <ant....@gmail.com> wrote: > Hi > I'm new to Spark. I'm doing some research to see if spark streaming can > solve my problem. I don't want to keep per-key state,b/c my data set is > very huge and keep a little longer time, it not viable to keep all per key > state in memory.Instead, i want to have a bloom filter based state. Does it > possible to achieve this in Spark streaming. > > -- > Anty Rao >
