Yeah, DirectKafkaInputDStream always returns a RDD even if it's empty. Feel free to send a PR to improve it.
On Thu, Feb 11, 2016 at 1:09 PM, Sebastian Piu <sebastian....@gmail.com> wrote: > I'm using the Kafka direct stream api but I can have a look on extending > it to have this behaviour > > Thanks! > On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <shixi...@databricks.com> > wrote: > >> Are you using a custom input dstream? If so, you can make the `compute` >> method return None to skip a batch. >> >> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <sebastian....@gmail.com> >> wrote: >> >>> I was wondering if there is there any way to skip batches with zero >>> events when streaming? >>> By skip I mean avoid the empty rdd from being created at all? >>> >> >>
