[
https://issues.apache.org/jira/browse/FLINK-33724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiaxing Chen updated FLINK-33724:
---------------------------------
Description:
When running a job in application mode, flink will save program arguments in
flink-conf.yaml. These arguments will be loaded when deploying an application
on YARN. However multiple lines argument cannot be loaded properly.
In my case, I submit a job with sql which is a multiple lines argument:
{code:java}
/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
In flink-conf.yaml，it saved as:
{code:java}
$internal.application.program-args： "INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
And produced some warning logs when loaded from flink-conf.yaml :
{code:java}
INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.application.program-args, "INSERT INTO tableA
SELECT a,
WARN org.apache.flink.configuration.GlobalConfiguration [] - Error
while trying to split key and value in configuration file /{working
dir}/flink-conf.yaml:{line num} "SELECT "
...{code}
I dig into the source code and find out that the reader and writer methods of
YAML file cannot properly parse YAML format. Maybe it would be better to use
some third-party tools, such as snakeyaml, to do this job.
was:
When running a job in application mode, program arguments are stored in the
flink-conf.yaml file. These arguments will be loaded when deploying an
application on YARN. However multiple lines argument cannot be loaded properly.
In my case, I submit a job with sql which is a multiple lines argument:
{code:java}
/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
In flink-conf.yaml，it saved as:
{code:java}
$internal.application.program-args： "INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
And produced some warning logs when loaded from flink-conf.yaml :
{code:java}
INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.application.program-args, "INSERT INTO tableA
SELECT a,
WARN org.apache.flink.configuration.GlobalConfiguration [] - Error
while trying to split key and value in configuration file /{working
dir}/flink-conf.yaml:{line num} "SELECT "
...{code}
I dig into the source code and find out that the reader and writer methods of
YAML file cannot properly parse YAML format. Maybe it would be better to use
some third-party tools, such as snakeyaml, to do this job.
> Application mode doesn't support multiple lines argument
> --------------------------------------------------------
>
> Key: FLINK-33724
> URL: https://issues.apache.org/jira/browse/FLINK-33724
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.18.0
> Reporter: Jiaxing Chen
> Priority: Major
>
> When running a job in application mode, flink will save program arguments in
> flink-conf.yaml. These arguments will be loaded when deploying an application
> on YARN. However multiple lines argument cannot be loaded properly.
> In my case, I submit a job with sql which is a multiple lines argument:
>
> {code:java}
> /bin/flink run-application
> -t yarn-application
> ...
> myjob.jar
> "INSERT INTO tableA
> SELECT
> a,
> b,
> c
> FROM tableB;"{code}
> In flink-conf.yaml，it saved as:
> {code:java}
> $internal.application.program-args： "INSERT INTO tableA
> SELECT
> a,
> b,
> c
> FROM tableB;"{code}
> And produced some warning logs when loaded from flink-conf.yaml :
> {code:java}
> INFO org.apache.flink.configuration.GlobalConfiguration [] -
> Loading configuration property: $internal.application.program-args, "INSERT
> INTO tableA SELECT a,
> WARN org.apache.flink.configuration.GlobalConfiguration [] - Error
> while trying to split key and value in configuration file /{working
> dir}/flink-conf.yaml:{line num} "SELECT "
> ...{code}
> I dig into the source code and find out that the reader and writer methods of
> YAML file cannot properly parse YAML format. Maybe it would be better to use
> some third-party tools, such as snakeyaml, to do this job.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
