Not built in, you're going to have to do some work.
On Sep 2, 2016 16:33, "sagarcasual ." <sagarcas...@gmail.com> wrote:
> Hi Cody, thanks for the reply.
> I am using Spark 1.6.1 with Kafka 0.9.
> When I want to stop streaming, stopping the context sounds ok, but for
> temporarily excluding partitions is there any way I can supply
> topic-partition info on the fly at the beginning of every pull dynamically.
> Will streaminglistener be of any help?
>
> On Fri, Sep 2, 2016 at 10:37 AM, Cody Koeninger <c...@koeninger.org>
> wrote:
>
>> If you just want to pause the whole stream, just stop the app and then
>> restart it when you're ready.
>>
>> If you want to do some type of per-partition manipulation, you're
>> going to need to write some code. The 0.10 integration makes the
>> underlying kafka consumer pluggable, so you may be able to wrap a
>> consumer to do what you need.
>>
>> On Fri, Sep 2, 2016 at 12:28 PM, sagarcasual . <sagarcas...@gmail.com>
>> wrote:
>> > Hello, this is for
>> > Pausing spark kafka streaming (direct) or exclude/include some
>> partitions on
>> > the fly per batch
>> > =========================================================
>> > I have following code that creates a direct stream using Kafka
>> connector for
>> > Spark.
>> >
>> > final JavaInputDStream<KafkaMessage> msgRecords =
>> > KafkaUtils.createDirectStream(
>> > jssc, String.class, String.class, StringDecoder.class,
>> > StringDecoder.class,
>> > KafkaMessage.class, kafkaParams, topicsPartitions,
>> > message -> {
>> > return KafkaMessage.builder()
>> > .
>> > .build();
>> > }
>> > );
>> >
>> > However I want to handle a situation, where I can decide that this
>> streaming
>> > needs to pause for a while on conditional basis, is there any way to
>> achieve
>> > this? Say my Kafka is undergoing some maintenance, so between 10AM to
>> 12PM
>> > stop processing, and then again pick up at 12PM from the last offset,
>> how do
>> > I do it?
>> >
>> > Also, assume all of a sudden we want to take one-or-more of the
>> partitions
>> > for a pull and add it back after some pulls, how do I achieve that?
>> >
>> > -Regards
>> > Sagar
>> >
>>
>
>
