mapWithState supports checkpoint. There has been some bug fix since release of 1.6.0 e.g. SPARK-12591 NullPointerException using checkpointed mapWithState with KryoSerializer
which is in the upcoming 1.6.1 Cheers On Sat, Feb 13, 2016 at 12:05 PM, Abhishek Anand <abhis.anan...@gmail.com> wrote: > Does mapWithState checkpoints the data ? > > When my application goes down and is restarted from checkpoint, will > mapWithState need to recompute the previous batches data ? > > Also, to use mapWithState I will need to upgrade my application as I am > using version 1.4.0 and mapWithState isnt supported there. Is there any > other work around ? > > Cheers!! > Abhi > > On Fri, Feb 12, 2016 at 3:33 AM, Sebastian Piu <sebastian....@gmail.com> > wrote: > >> Looks like mapWithState could help you? >> On 11 Feb 2016 8:40 p.m., "Abhishek Anand" <abhis.anan...@gmail.com> >> wrote: >> >>> Hi All, >>> >>> I have an use case like follows in my production environment where I am >>> listening from kafka with slideInterval of 1 min and windowLength of 2 >>> hours. >>> >>> I have a JavaPairDStream where for each key I am getting the same key >>> but with different value,which might appear in the same batch or some next >>> batch. >>> >>> When the key appears second time I need to update a field in value of >>> previous key with a field in the later key. The keys for which the >>> combination keys do not come should be rejected after 2 hours. >>> >>> At the end of each second I need to output the result to external >>> database. >>> >>> For example : >>> >>> Suppose valueX is object of MyClass with fields int a, String b >>> At t=1sec I am getting >>> key0,value0(0,"prev0") >>> key1,value1 (1, "prev1") >>> key2,value2 (2,"prev2") >>> key2,value3 (3, "next2") >>> >>> Output to database after 1 sec >>> key2, newValue (2,"next2") >>> >>> At t=2 sec getting >>> key3,value4(4,"prev3") >>> key1,value5(5,"next1") >>> >>> Output to database after 2 sec >>> key1,newValue(1,"next1") >>> >>> At t=3 sec >>> key4,value6(6,"prev4") >>> key3,value7(7,"next3") >>> key5,value5(8,"prev5") >>> key5,value5(9,"next5") >>> key0,value0(10,"next0") >>> >>> Output to database after 3 sec >>> key0,newValue(0,"next0") >>> key3,newValue(4,"next3") >>> key5,newValue(8,"next5") >>> >>> >>> Please suggest how this can be achieved. >>> >>> >>> Thanks a lot !!!! >>> Abhi >>> >>> >>> >
