Those jobs will still be created for each valid time, they just may not have many messages in them
On Mon, Aug 3, 2015 at 11:11 PM, Shushant Arora <[email protected]> wrote: > 1.In spark 1.3(Non receiver) - If my batch interval is 1 sec and I don't > set spark.streaming.kafka.maxRatePerPartition - so default behavious is to > bring all messages from kafka from last offset to current offset ? > > Say no of messages were large and it took 5 sec to process those so will > all jobs for interval 2-5 sec be queued and created afterwards or should > not be created since all messages are already processed for those interval > also? > > 2.In spark streaming 1.2(Receiver based) if I don't set > spark.streaming.receiver.maxRate > - will it consume all messages from last offset or it will just consume > messages whatever it can consume in this batch interval of 1 sec. >
