Hello, this is for
Pausing spark kafka streaming (direct) or exclude/include some partitions
on the fly per batch
=========================================================
I have following code that creates a direct stream using Kafka connector
for Spark.

final JavaInputDStream<KafkaMessage> msgRecords =
KafkaUtils.createDirectStream(
            jssc, String.class, String.class, StringDecoder.class,
StringDecoder.class,
            KafkaMessage.class, kafkaParams, topicsPartitions,
            message -> {
                return KafkaMessage.builder()
                        .
                        .build();
            }
    );

However I want to handle a situation, where I can decide that this
streaming needs to pause for a while on conditional basis, is there any way
to achieve this? Say my Kafka is undergoing some maintenance, so between
10AM to 12PM stop processing, and then again pick up at 12PM from the last
offset, how do I do it?

Also, assume all of a sudden we want to take one-or-more of the partitions
for a pull and add it back after some pulls, how do I achieve that?

-Regards
Sagar

Reply via email to