Re: UUID on TableAPI

2022-04-26 Thread Martijn Visser 
Hi Ian,

What is the problem that you're trying to solve?

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser


On Tue, 26 Apr 2022 at 02:41, lan tran  wrote:

> Ok, thanks for the clarification.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>
> *From: *Francis Conroy 
> *Sent: *Tuesday, April 26, 2022 7:26 AM
> *To: *lan tran 
> *Cc: *user@flink.apache.org
> *Subject: *Re: UUID on TableAPI
>
>
>
> Hi Quynh,
>
>
>
> My understanding is mostly based on the documentation I linked in the
> first reply. If the flink version and the query both remain the same then
> you can restart a job from a savepoint, this means that it might be
> workable for running a low-criticality job on say an AWS spot instance.
> That's about all I know.
>
>
>
> On Tue, 26 Apr 2022 at 10:17, lan tran  wrote:
>
> Hi Francis,
>
> Thanks for the reply. However, can you elaborate more on the part ‘work
> for cases where you wish to pause/resume a job’ ? Is there another way
> besides savepoint having this mechanism that can implement in Table API ?
>
> Best,
> Quynh
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>
> *From: *Francis Conroy 
> *Sent: *Tuesday, April 26, 2022 7:07 AM
> *To: *lan tran 
> *Cc: *user@flink.apache.org
> *Subject: *Re: UUID on TableAPI
>
>
>
> Hi  Quynh,
>
>
>
> Set the savepoint UUID? Is that what you mean? The savepoint UUIDs are
> issued dynamically when you request them, flink won't know automatically
> what the last savepoint was, but you can start a new job and restore from a
> savepoint by passing in the UUID. All that said there are limitations
> around using savepoints and Flink SQL because of the way the planner works
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution.
> However it might work for cases where you wish to pause/resume a job.
>
>
>
> On Fri, 22 Apr 2022 at 13:54, lan tran  wrote:
>
> Hi team,
> Currently, I want to use savepoints in Flink. However, one of the things
> that I concern is that is there any way we can set the UUID while using
> Table API (SQL API) ? If not, does it has any mechanism to know that when
> we start the Flink again, it will know that it was that UUID ?
>
> Best,
> Quynh.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>
>
>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
>
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>
>
>
>
>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
>
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>
>
>

RE: UUID on TableAPI

2022-04-25 Thread lan tran 
Ok, thanks for the clarification.   Sent from Mail for Windows From: Francis ConroySent: Tuesday, April 26, 2022 7:26 AMTo: lan tranCc: user@flink.apache.orgSubject: Re: UUID on TableAPI Hi Quynh,  My understanding is mostly based on the documentation I linked in the first reply. If the flink version and the query both remain the same then you can restart a job from a savepoint, this means that it might be workable for running a low-criticality job on say an AWS spot instance. That's about all I know. On Tue, 26 Apr 2022 at 10:17, lan tran <indigoblue7...@gmail.com> wrote:Hi Francis,Thanks for the reply. However, can you elaborate more on the part ‘work for cases where you wish to pause/resume a job’ ? Is there another way besides savepoint having this mechanism that can implement in Table API ? Best,Quynh  Sent from Mail for Windows From: Francis ConroySent: Tuesday, April 26, 2022 7:07 AMTo: lan tranCc: user@flink.apache.orgSubject: Re: UUID on TableAPI Hi  Quynh, Set the savepoint UUID? Is that what you mean? The savepoint UUIDs are issued dynamically when you request them, flink won't know automatically what the last savepoint was, but you can start a new job and restore from a savepoint by passing in the UUID. All that said there are limitations around using savepoints and Flink SQL because of the way the planner works https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution. However it might work for cases where you wish to pause/resume a job. On Fri, 22 Apr 2022 at 13:54, lan tran <indigoblue7...@gmail.com> wrote:Hi team,Currently, I want to use savepoints in Flink. However, one of the things that I concern is that is there any way we can set the UUID while using Table API (SQL API) ? If not, does it has any mechanism to know that when we start the Flink again, it will know that it was that UUID ?Best,Quynh. Sent from Mail for Windows  This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia  This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia 

Re: UUID on TableAPI

2022-04-25 Thread Francis Conroy 
 Hi Quynh,

My understanding is mostly based on the documentation I linked in the first
reply. If the flink version and the query both remain the same then you can
restart a job from a savepoint, this means that it might be workable for
running a low-criticality job on say an AWS spot instance. That's about all
I know.

On Tue, 26 Apr 2022 at 10:17, lan tran  wrote:

> Hi Francis,
>
> Thanks for the reply. However, can you elaborate more on the part ‘work
> for cases where you wish to pause/resume a job’ ? Is there another way
> besides savepoint having this mechanism that can implement in Table API ?
>
> Best,
> Quynh
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>
> *From: *Francis Conroy 
> *Sent: *Tuesday, April 26, 2022 7:07 AM
> *To: *lan tran 
> *Cc: *user@flink.apache.org
> *Subject: *Re: UUID on TableAPI
>
>
>
> Hi  Quynh,
>
>
>
> Set the savepoint UUID? Is that what you mean? The savepoint UUIDs are
> issued dynamically when you request them, flink won't know automatically
> what the last savepoint was, but you can start a new job and restore from a
> savepoint by passing in the UUID. All that said there are limitations
> around using savepoints and Flink SQL because of the way the planner works
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution.
> However it might work for cases where you wish to pause/resume a job.
>
>
>
> On Fri, 22 Apr 2022 at 13:54, lan tran  wrote:
>
> Hi team,
> Currently, I want to use savepoints in Flink. However, one of the things
> that I concern is that is there any way we can set the UUID while using
> Table API (SQL API) ? If not, does it has any mechanism to know that when
> we start the Flink again, it will know that it was that UUID ?
>
> Best,
> Quynh.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>
>
>
> This email and any attachments are proprietary and confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
> received this email in error, please let us know immediately by reply email
> and delete it from your system. You may not use, disseminate, distribute or
> copy this message nor disclose its contents to anyone.
>
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300
> Australia
>
>
>

-- 
This email and any attachments are proprietary and confidential and are 
intended solely for the use of the individual to whom it is addressed. Any 
views or opinions expressed are solely those of the author and do not 
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have 
received this email in error, please let us know immediately by reply email 
and delete it from your system. You may not use, disseminate, distribute or 
copy this message nor disclose its contents to anyone. 
SwitchDin Pty Ltd 
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia

RE: UUID on TableAPI

2022-04-25 Thread lan tran 
Hi Francis,Thanks for the reply. However, can you elaborate more on the part ‘work for cases where you wish to pause/resume a job’ ? Is there another way besides savepoint having this mechanism that can implement in Table API ? Best,Quynh  Sent from Mail for Windows From: Francis ConroySent: Tuesday, April 26, 2022 7:07 AMTo: lan tranCc: user@flink.apache.orgSubject: Re: UUID on TableAPI Hi  Quynh, Set the savepoint UUID? Is that what you mean? The savepoint UUIDs are issued dynamically when you request them, flink won't know automatically what the last savepoint was, but you can start a new job and restore from a savepoint by passing in the UUID. All that said there are limitations around using savepoints and Flink SQL because of the way the planner works https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution. However it might work for cases where you wish to pause/resume a job. On Fri, 22 Apr 2022 at 13:54, lan tran <indigoblue7...@gmail.com> wrote:Hi team,Currently, I want to use savepoints in Flink. However, one of the things that I concern is that is there any way we can set the UUID while using Table API (SQL API) ? If not, does it has any mechanism to know that when we start the Flink again, it will know that it was that UUID ?Best,Quynh. Sent from Mail for Windows  This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia 

Re: UUID on TableAPI

2022-04-25 Thread Francis Conroy 
Hi  Quynh,

Set the savepoint UUID? Is that what you mean? The savepoint UUIDs are
issued dynamically when you request them, flink won't know automatically
what the last savepoint was, but you can start a new job and restore from a
savepoint by passing in the UUID. All that said there are limitations
around using savepoints and Flink SQL because of the way the planner works
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution.
However it might work for cases where you wish to pause/resume a job.

On Fri, 22 Apr 2022 at 13:54, lan tran  wrote:

> Hi team,
> Currently, I want to use savepoints in Flink. However, one of the things
> that I concern is that is there any way we can set the UUID while using
> Table API (SQL API) ? If not, does it has any mechanism to know that when
> we start the Flink again, it will know that it was that UUID ?
>
> Best,
> Quynh.
>
>
>
> Sent from Mail  for
> Windows
>
>
>

-- 
This email and any attachments are proprietary and confidential and are 
intended solely for the use of the individual to whom it is addressed. Any 
views or opinions expressed are solely those of the author and do not 
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have 
received this email in error, please let us know immediately by reply email 
and delete it from your system. You may not use, disseminate, distribute or 
copy this message nor disclose its contents to anyone. 
SwitchDin Pty Ltd 
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia

UUID on TableAPI

2022-04-21 Thread lan tran 
Hi team,Currently, I want to use savepoints in Flink. However, one of the things that I concern is that is there any way we can set the UUID while using Table API (SQL API) ? If not, does it has any mechanism to know that when we start the Flink again, it will know that it was that UUID ?Best,Quynh. Sent from Mail for Windows