Looks like mapWithState could help you? On 11 Feb 2016 8:40 p.m., "Abhishek Anand" <abhis.anan...@gmail.com> wrote:
> Hi All, > > I have an use case like follows in my production environment where I am > listening from kafka with slideInterval of 1 min and windowLength of 2 > hours. > > I have a JavaPairDStream where for each key I am getting the same key but > with different value,which might appear in the same batch or some next > batch. > > When the key appears second time I need to update a field in value of > previous key with a field in the later key. The keys for which the > combination keys do not come should be rejected after 2 hours. > > At the end of each second I need to output the result to external database. > > For example : > > Suppose valueX is object of MyClass with fields int a, String b > At t=1sec I am getting > key0,value0(0,"prev0") > key1,value1 (1, "prev1") > key2,value2 (2,"prev2") > key2,value3 (3, "next2") > > Output to database after 1 sec > key2, newValue (2,"next2") > > At t=2 sec getting > key3,value4(4,"prev3") > key1,value5(5,"next1") > > Output to database after 2 sec > key1,newValue(1,"next1") > > At t=3 sec > key4,value6(6,"prev4") > key3,value7(7,"next3") > key5,value5(8,"prev5") > key5,value5(9,"next5") > key0,value0(10,"next0") > > Output to database after 3 sec > key0,newValue(0,"next0") > key3,newValue(4,"next3") > key5,newValue(8,"next5") > > > Please suggest how this can be achieved. > > > Thanks a lot !!!! > Abhi > > >
