Thanks for taking a look.
Are you saying that the longer term direction is to get rid of the execute
method from StreamExecutionEnvironment and instead construct the cluster
client outside?
That would currently expose even more internals to the user. Considering
the current implementation in RemoteStreamEnvironment:
@Override
public JobExecutionResult execute(String jobName) throws
ProgramInvocationException {
StreamGraph streamGraph = getStreamGraph();
streamGraph.setJobName(jobName);
transformations.clear();
return executeRemotely(streamGraph, jarFiles);
}
We would use
env.getStreamGraph().getJobGraph().setSavepointRestoreSettings(..) in the
Beam code and then use the cluster client directly.
If we wanted to keep this hidden from users, we could add
setSavePointRestoreSettings to RemoteStreamEnvironment and
LocalStreamEnvironment and deal with it internally.
Alternatively, the remote environment could serve just as cluster client
factory.
WDYT?
On Thu, Nov 29, 2018 at 2:35 AM Chesnay Schepler <ches...@apache.org> wrote:
> I'm not aware of any plans to expose this in the
> StreamExecutionEnvironment.
>
> The issue would be that we would start mixing submission details with the
> job definition, which results in redundancy and weird semantics, e.g.,
> which savepoint configuration takes priority if both a job and CLI job
> submission specify it?
>
> Looking at the Beam JIRA it would be sufficient to have this in the
> RemoteStreamEnvironment (which would be *less *problematic since the
> issue above is baked into this class anyway), however I would recommend
> migrating to a ClusterClient for these use-cases.
>
> On 29.11.2018 08:18, Thomas Weise wrote:
>
> Hi,
>
> Currently it is not possible to submit a job with savepoint restore option
> through the execution environment. I found that while attempting to add the
> support to the Flink runner in Beam
> (https://issues.apache.org/jira/browse/BEAM-5396)
>
> I also found https://issues.apache.org/jira/browse/FLINK-9644 - but is
> there a plan to support restore from savepoint
> through StreamExecutionEnvironment in general?
>
> Thanks,
> Thomas
>
>
>
>
