Re: [Spark Structued Streaming]: Read kafka offset from a timestamp

2018-11-19 Thread Jungtaek Lim
It really depends on whether we use it only for starting query (instead of restoring from checkpoint) or we would want to restore the previous batch from specific time (with restoring state as well). For former would make sense and I'll try to see whether I can address it. For latter that doesn't

[Spark Structued Streaming]: Read kafka offset from a timestamp

2018-11-17 Thread puneetloya
I would like to request a feature for reading data from Kafka Source based on a timestamp. So that if the application needs to process data from a certain time, it should be able to do it. I do agree, that there is checkpoint which gives us a continuation of stream process but what if I want to