Hi Chesnay,
That is an interesting proposal, thank you!
I was doing something similar with the OutputFormat#close() method respecting
the Format's parallelism. Using FinalizeOnMaster will make things easier.
But the problem is that several OutputFormats must be synchronized externally -
every
This goes in the right direction; have a look at
org.apache.flink.api.common.io.FinalizeOnMaster, which an OutputFormat
can implement to run something on the Master after all subtasks have
been closed.
On 08/06/2020 17:25, Andrey Zagrebin wrote:
Hi Mark,
I do not know how you output the
I usually run some code after env.execute(), it's not elegant but it works
(only if you run the code from Cli client, not from REST one)
On Mon, Jun 8, 2020 at 5:25 PM Andrey Zagrebin wrote:
> Hi Mark,
>
> I do not know how you output the results in your pipeline.
> If you use
Hi Mark,
I do not know how you output the results in your pipeline.
If you use DataSet#output(OutputFormat outputFormat), you could try to
extend the format with a custom close method which should be called once
the task of the sink batch operator is done in the task manager.
I also cc Aljoscha,
Hi Jeff,
Unfortunately this is not good enough for me.
My clients are very volatile, they start a batch and can go away any moment
without waiting for it to finish. Think of an elastic web application or an AWS
Lambda.
I hopped to find something what could be deployed to the cluster together
It would run in the client side where ExecutionEnvironment is created.
Mark Davis 于2020年6月6日周六 下午8:14写道：
> Hi Jeff,
>
> Thank you very much! That is exactly what I need.
>
> Where the listener code will run in the cluster deployment(YARN, k8s)?
> Will it be sent over the network?
>
> Thank you!
Hi Jeff,
Thank you very much! That is exactly what I need.
Where the listener code will run in the cluster deployment(YARN, k8s)?
Will it be sent over the network?
Thank you!
Mark
‐‐‐ Original Message ‐‐‐
On Friday, June 5, 2020 6:13 PM, Jeff Zhang wrote:
> You can try JobListener
You can try JobListener which you can register to ExecutionEnvironment.
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/execution/JobListener.java
Mark Davis 于2020年6月6日周六 上午12:00写道：
> Hi there,
>
> I am running a Batch job with several outputs.
> Is
Hi there,
I am running a Batch job with several outputs.
Is there a way to run some code(e.g. release a distributed lock) after all
outputs are finished?
Currently I do this in a try-finally block around
ExecutionEnvironment.execute() call, but I have to switch to the detached
execution mode
