Hi Priya and Vincent
Thank you for your reply!
It looks the new feature is implemented only in the latest version.
But I'm using Spark 2.3.0 so, in my understanding, I need to stop and
reload apps.
Thanks
On 2018/12/19 9:09, vincent gromakowski wrote:
I totally missed this new feature. Thanks for the pointer
Le mar. 18 déc. 2018 à 21:18, Priya Matpadi <[email protected]
<mailto:[email protected]>> a écrit :
Changes in streaming query that allow or disallow recovery from
checkpoint is clearly provided in
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovery-semantics-after-changes-in-a-streaming-query.
On Tue, Dec 18, 2018 at 9:45 AM vincent gromakowski
<[email protected]
<mailto:[email protected]>> wrote:
Checkpointing is only used for failure recovery not for app
upgrades. You need to manually code the unload/load and save it
to a persistent store
Le mar. 18 déc. 2018 à 17:29, Priya Matpadi <[email protected]
<mailto:[email protected]>> a écrit :
Using checkpointing for graceful updates is my understanding
as well, based on the writeup in
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing,
and some prototyping. Have you faced any missed events?
On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa
<[email protected]
<mailto:[email protected]>> wrote:
Hi
Now I'm trying to update my structured streaming
application.
But I have no idea how to update it gracefully.
Should I stop it, replace a jar file then restart it?
In my understanding, in that case, all the state will be
recovered if I
use checkpoints.
Is this correct?
Thank you,
--
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
<mailto:[email protected]>
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
