Spark Streaming - Latest batch-time can't keep up with current time

2016-01-18 Thread Collin Shi
Hi all, After having submit the job, the latest batch-time is almost as same as current time at first. Let's say, if current time is '12:00:00', then the latest batch-time would be '11:59:59'. But as time goes, the difference is getting greater and greater. For instance , current time is

[Spark Streaming] "Could not compute split, block input-0-1452563923800 not found” when trying to recover from checkpoint data

2016-01-13 Thread Collin Shi
Hi I was doing a simple updateByKey transformation and print on the data received from socket, and spark version is 1.4.0. The first submit went all right, but after I kill (CTRL + C) the job and submit again. Apparently spark was trying to recover from the checkpoint data , but then the