Re: How to know whether I'm in the first batch of spark streaming

2016-04-21 Thread Yu Xie
are Labs > > - > "Courage doesn't always roar. Sometimes courage is the quiet voice at the > end of the day saying I will try again" > > > > From:Yu Xie <yuu...@

How to know whether I'm in the first batch of spark streaming

2016-04-19 Thread Yu Xie
hi spark users I'm running a spark streaming application, with concurrentJobs > 1, so maybe more than one batches could run together. Now I would like to do some init work in the first batch based on the "time" of the first batch. So even the second batch runs faster than the first batch, I

can checkpoint and write ahead log save the data in queued batch?

2016-03-11 Thread Yu Xie
Hi spark user I am running an spark streaming app that use receiver from a pubsub system, and the pubsub system does NOT support ack. And I don't want the data to be lost if there is a driver failure, and by accident, the batches queue up at that time. I tested by generating some queued