If you just want to pause the whole stream, just stop the app and then
restart it when you're ready.
If you want to do some type of per-partition manipulation, you're
going to need to write some code. The 0.10 integration makes the
underlying kafka consumer pluggable, so you may be able to wrap a
consumer to do what you need.
On Fri, Sep 2, 2016 at 12:28 PM, sagarcasual . <sagarcas...@gmail.com> wrote:
> Hello, this is for
> Pausing spark kafka streaming (direct) or exclude/include some partitions on
> the fly per batch
> =========================================================
> I have following code that creates a direct stream using Kafka connector for
> Spark.
>
> final JavaInputDStream<KafkaMessage> msgRecords =
> KafkaUtils.createDirectStream(
> jssc, String.class, String.class, StringDecoder.class,
> StringDecoder.class,
> KafkaMessage.class, kafkaParams, topicsPartitions,
> message -> {
> return KafkaMessage.builder()
> .
> .build();
> }
> );
>
> However I want to handle a situation, where I can decide that this streaming
> needs to pause for a while on conditional basis, is there any way to achieve
> this? Say my Kafka is undergoing some maintenance, so between 10AM to 12PM
> stop processing, and then again pick up at 12PM from the last offset, how do
> I do it?
>
> Also, assume all of a sudden we want to take one-or-more of the partitions
> for a pull and add it back after some pulls, how do I achieve that?
>
> -Regards
> Sagar
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
