Hi All, I have an use case like follows in my production environment where I am listening from kafka with slideInterval of 1 min and windowLength of 2 hours.
I have a JavaPairDStream where for each key I am getting the same key but with different value,which might appear in the same batch or some next batch. When the key appears second time I need to update a field in value of previous key with a field in the later key. The keys for which the combination keys do not come should be rejected after 2 hours. At the end of each second I need to output the result to external database. For example : Suppose valueX is object of MyClass with fields int a, String b At t=1sec I am getting key0,value0(0,"prev0") key1,value1 (1, "prev1") key2,value2 (2,"prev2") key2,value3 (3, "next2") Output to database after 1 sec key2, newValue (2,"next2") At t=2 sec getting key3,value4(4,"prev3") key1,value5(5,"next1") Output to database after 2 sec key1,newValue(1,"next1") At t=3 sec key4,value6(6,"prev4") key3,value7(7,"next3") key5,value5(8,"prev5") key5,value5(9,"next5") key0,value0(10,"next0") Output to database after 3 sec key0,newValue(0,"next0") key3,newValue(4,"next3") key5,newValue(8,"next5") Please suggest how this can be achieved. Thanks a lot !!!! Abhi
