Re: Spark streaming batch time displayed is not current system time but it is processing current messages
Using http://www.ruddwire.com/handy-code/date-to-millisecond-calculators/#.VxZh3iMrKuo , 1460823008000 is shown to be 'Sat Apr 16 2016 09:10:08 GMT-0700' Can you clarify the 4 day difference ? bq. 'right now April 14th' The date of your email was Apr 16th. On Sat, Apr 16, 2016 at 9:39 AM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > Can anyone help me in debugging this issue please. > > > On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A < > hemalatha.amru...@googlemail.com> wrote: > >> Hi, >> >> I am facing a problem in Spark streaming. >> > > > Time: 1460823006000 ms > --- > > --- > Time: 1460823008000 ms > --- > > > > >> The time displayed in Spark streaming console as above is 4 days prior >> i.e., April 10th, which is not current system time of the cluster but the >> job is processing current messages that is pushed right now April 14th. >> >> Can anyone please advice what time does Spark streaming display? Also, >> when there is scheduling delay of say 8 hours, what time does Spark >> display- current rime or hours behind? >> >> -- >> >> >> Regards >> Hemalatha >> > > > > -- > > > Regards > Hemalatha >
Re: Spark streaming batch time displayed is not current system time but it is processing current messages
This can happen if system time is not in sync. By default, streaming uses SystemClock(it also supports ManualClock) and that relies on System.currentTimeMillis() for determining start time. Prashant Sharma On Sat, Apr 16, 2016 at 10:09 PM, Hemalatha A < hemalatha.amru...@googlemail.com> wrote: > Can anyone help me in debugging this issue please. > > > On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A < > hemalatha.amru...@googlemail.com> wrote: > >> Hi, >> >> I am facing a problem in Spark streaming. >> > > > Time: 1460823006000 ms > --- > > --- > Time: 1460823008000 ms > --- > > > > >> The time displayed in Spark streaming console as above is 4 days prior >> i.e., April 10th, which is not current system time of the cluster but the >> job is processing current messages that is pushed right now April 14th. >> >> Can anyone please advice what time does Spark streaming display? Also, >> when there is scheduling delay of say 8 hours, what time does Spark >> display- current rime or hours behind? >> >> -- >> >> >> Regards >> Hemalatha >> > > > > -- > > > Regards > Hemalatha >
