It's at the time compute() gets called, which should be near the time the batch should have been queued.
On Tue, Sep 1, 2015 at 8:02 AM, Shushant Arora <shushantaror...@gmail.com> wrote: > Hi > > In spark streaming 1.3 with kafka- when does driver bring latest offsets > of this run - at start of each batch or at time when batch gets queued ? > > Say few of my batches take longer time to complete than their batch > interval. So some of batches will go in queue. Will driver waits for > queued batches to get started or just brings the latest offsets before > they even actually started. And when they start running they will work on > old offsets brought at time when they were queued. > >
