[
https://issues.apache.org/jira/browse/FLINK-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440151#comment-17440151
]
Caizhi Weng edited comment on FLINK-24757 at 11/8/21, 7:13 AM:
---------------------------------------------------------------
This issue is caused by the mismatch of {{table.dml-sync}} (whose default value
is false) and {{execution.attached}} (whose default value is true, if submitted
from SQL client).
According to the offline discussion with [~twalthr], these two options are
orthogonal. {{execution.attached}} determines if cluster needs to wait for
client to fetch the result or just leaves by itself; {{table.dml-sync}}
determines if client will wait for and fetch that result.
However they are somewhat related. If client doesn’t want that result but the
cluster is willing to wait then it will have to wait forever; On the other hand
if client needs that result but cluster leaves by itself there will be an
exception.
Currently we don't have a good solution to the case when user specify the two
options at the same time, so we're just fixing it in SQL client submission and
improve the documentation for {{{}table.dml-sync{}}}. Let's come back to this
issue when a {{flink run}} + SQL user complains about this.
P.S.: The default value of {{execution.attached}} itself is false. However SQL
client is using some {{ProgramOptions}} thingy (see
{{DefaultContext#createExecutionConfig}}) in which command line options will be
read and if "-d" is found then it will be set to detach mode. But SQL client is
not passing any command line arguments to {{ProgramOptions}} (see
{{DefaultContext#new}}, it passes an empty string array) so it defaults to
attach mode.
was (Author: tsreaper):
This issue is caused by the mismatch of {{table.dml-sync}} (whose default value
is false) and {{execution.attached}} (whose default value is true).
According to the offline discussion with [~twalthr], these two options are
orthogonal. {{execution.attached}} determines if cluster needs to wait for
client to fetch the result or just leaves by itself; {{table.dml-sync}}
determines if client will wait for and fetch that result.
However they are somewhat related. If client doesn’t want that result but the
cluster is willing to wait then it will have to wait forever; On the other hand
if client needs that result but cluster leaves by itself there will be an
exception.
Currently we don't have a good solution to the case when user specify the two
options at the same time, so we're just fixing it in SQL client submission and
improve the documentation for {{table.dml-sync}}. Let's come back to this issue
when a {{flink run}} + SQL user complains about this.
> Yarn application is not terminated after the job finishes when submitting a
> yarn-per-job insert job with SQL client
> -------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-24757
> URL: https://issues.apache.org/jira/browse/FLINK-24757
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
> Affects Versions: 1.14.0
> Reporter: Caizhi Weng
> Priority: Major
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> I've seen this problem for about three times in the user mailing thread
> (previously I suspect that the users are specifying the wrong
> {{{}execution.target{}}}) until I myself also bumped into this problem. I've
> submitted a yarn-per-job batch insert SQL with Flink SQL client and after the
> job finishes Yarn application is not terminated.
> This is because yarn job cluster is using {{MiniDispatcher}} and it will
> directly terminate only in detached execution mode. This execution mode is
> (through some function calls) related to {{DeploymentOptions#ATTACHED}} which
> is true by default if jobs are submitted from SQL client.
> When submitting an insert job, SQL client will not wait for the job to
> finish. Instead it only reports the job id. So I think it is reasonable to
> set detached mode for every insert job.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
