Any updates on how can I use ManualClock other than editing the Spark source code?
On Wed, Mar 1, 2017 at 10:19 AM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > It is certainly possible through a hack. > I was referring to below post where TD says it is possible thru a hack. I > wanted to know if there is any way other than editing the Spark source > code. > > https://groups.google.com/forum/#!searchin/spark-users/manua > lclock%7Csort:relevance/spark-users/ES8X1l_xn5s/6PvGGRDfgnMJ > > On Wed, Mar 1, 2017 at 7:09 AM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> I don't think using ManualClock is a right way to fix your problem here >> in Spark Streaming. >> >> ManualClock in Spark is mainly used for unit test, it should manually >> advance the time to make the unit test work. The usage looks different >> compared to the scenario you mentioned. >> >> Thanks >> Jerry >> >> On Tue, Feb 28, 2017 at 10:53 PM, Hemalatha A < >> hemalatha.amru...@googlemail.com> wrote: >> >>> >>> Hi, >>> >>> I am running streaming application reading data from kafka and >>> performing window operations on it. I have a usecase where all incoming >>> events have a fixed latency of 10s, which means data belonging to minute >>> 10:00:00 will arrive 10s late at 10:00:10. >>> >>> I want to set the spark clock to "Manualclock" and set the time behind >>> by 10s so that the batch calculation triggers at 10:00:10, during which >>> time all the events for the previous minute has arrived. >>> >>> But, I see that "spark.streaming.clock" is hardcoded to " >>> org.apache.spark.util.SystemClock" in the code. >>> >>> Is there a way to easily hack this property to use Manual clock. >>> -- >>> >>> >>> Regards >>> Hemalatha >>> >> >> > > > -- > > > Regards > Hemalatha > -- Regards Hemalatha
