>>>>> foreachBatch(*sendToControl*). \
>>>>>> trigger(processingTime='2 seconds'). \
>>>>>> queryName('newtopic'). \
>>>>>>
>>>>>> That method sendToControl does what is needed
>>>&
queue = dfnewtopic.select(col("queue")).collect()[0][0]
>>>>> status = dfnewtopic.select(col("status")).collect()[0][0]
>>>>>
>>>>> if((queue == 'md')) & (status == 'false')):
>>>>>
ame
>>>> if(name == 'md'):
>>>> print(f"""Terminating streaming process {name}""")
>>>> e.stop()
>>>> else:
>>>> print("DataFrame newt
-- Forwarded message -
From: Mich Talebzadeh
Date: Thu, 6 May 2021 at 20:07
Subject: Re: Graceful shutdown SPARK Structured Streaming
To: ayan guha
Cc: Gourav Sengupta , user @spark <
user@spark.apache.org>
That is a valid question and I am not aware of any new addit
This is the output
>>>
>>> Terminating streaming process md
>>> wrote to DB ## this is the flag I added to ensure the current
>>> micro-bath was completed
>>> 2021-04-23 09:59:18,029 ERROR streaming.MicroBatchExecution: Query md
>>> [id
, runId =
>> 2ae55673-6bc2-4dbe-af60-9fdc0447bff5] terminated with error
>>
>> The various termination processes are described in
>>
>> Structured Streaming Programming Guide - Spark 3.1.1 Documentation
>> (apache.org)
>> <http://spark.apache.org/do
2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liab
5 May 2021 at 17:30, Gourav Sengupta <
gourav.sengupta.develo...@gmail.com> wrote:
> Hi,
>
> just thought of reaching out once again and seeking out your kind help to
> find out what is the best way to stop SPARK streaming gracefully. Do we
> still use the methods of creating a
,
Gourav Sengupta
-- Forwarded message -
From: Gourav Sengupta
Date: Wed, Apr 21, 2021 at 10:06 AM
Subject: Graceful shutdown SPARK Structured Streaming
To:
Dear friends,
is there any documentation available for gracefully stopping SPARK
Structured Streaming in 3.1.x?
I am
Dear friends,
is there any documentation available for gracefully stopping SPARK
Structured Streaming in 3.1.x?
I am referring to articles which are 4 to 5 years old and was wondering
whether there is a better way available today to gracefully shutdown a
SPARK streaming job.
Thanks a ton in adva
10 matches
Mail list logo
