[ 
https://issues.apache.org/jira/browse/FLINK-33724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiaxing Chen updated FLINK-33724:
---------------------------------
    Description: 
When running a job in application mode, flink will save program arguments in 
flink-conf.yaml. These arguments will be loaded when deploying an application 
on YARN. However multiple lines argument cannot be loaded properly.

In my case, I submit a job with sql which is a multiple lines argument:

 
{code:java}
/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c 
FROM tableB;"{code}
In flink-conf.yaml,it saved as:
{code:java}
$internal.application.program-args: "INSERT INTO tableA 
SELECT 
a,
b,
c 
FROM tableB;"{code}
And produced some warning logs when loaded from flink-conf.yaml :
{code:java}
INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading 
configuration property: $internal.application.program-args, "INSERT INTO tableA 
SELECT a,
WARN  org.apache.flink.configuration.GlobalConfiguration           [] - Error 
while trying to split key and value in configuration file /{working 
dir}/flink-conf.yaml:{line num} "SELECT "
...{code}
I dig into the source code and find out that the reader and writer methods of 
YAML file cannot properly parse YAML format. Maybe it would be better to use 
some third-party tools, such as snakeyaml, to do this job.

 

  was:
When running a job in application mode, program arguments  are stored in the 
flink-conf.yaml file. These arguments will be loaded when deploying an 
application on YARN. However multiple lines argument cannot be loaded properly.

In my case, I submit a job with sql which is a multiple lines argument:

 
{code:java}
/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c 
FROM tableB;"{code}
In flink-conf.yaml,it saved as:
{code:java}
$internal.application.program-args: "INSERT INTO tableA 
SELECT 
a,
b,
c 
FROM tableB;"{code}
And produced some warning logs when loaded from flink-conf.yaml :
{code:java}
INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading 
configuration property: $internal.application.program-args, "INSERT INTO tableA 
SELECT a,
WARN  org.apache.flink.configuration.GlobalConfiguration           [] - Error 
while trying to split key and value in configuration file /{working 
dir}/flink-conf.yaml:{line num} "SELECT "
...{code}
I dig into the source code and find out that the reader and writer methods of 
YAML file cannot properly parse YAML format. Maybe it would be better to use 
some third-party tools, such as snakeyaml, to do this job.

 


> Application mode doesn't support multiple lines argument
> --------------------------------------------------------
>
>                 Key: FLINK-33724
>                 URL: https://issues.apache.org/jira/browse/FLINK-33724
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN
>    Affects Versions: 1.18.0
>            Reporter: Jiaxing Chen
>            Priority: Major
>
> When running a job in application mode, flink will save program arguments in 
> flink-conf.yaml. These arguments will be loaded when deploying an application 
> on YARN. However multiple lines argument cannot be loaded properly.
> In my case, I submit a job with sql which is a multiple lines argument:
>  
> {code:java}
> /bin/flink run-application
> -t yarn-application
> ...
> myjob.jar
> "INSERT INTO tableA
> SELECT
> a,
> b,
> c 
> FROM tableB;"{code}
> In flink-conf.yaml,it saved as:
> {code:java}
> $internal.application.program-args: "INSERT INTO tableA 
> SELECT 
> a,
> b,
> c 
> FROM tableB;"{code}
> And produced some warning logs when loaded from flink-conf.yaml :
> {code:java}
> INFO  org.apache.flink.configuration.GlobalConfiguration           [] - 
> Loading configuration property: $internal.application.program-args, "INSERT 
> INTO tableA SELECT a,
> WARN  org.apache.flink.configuration.GlobalConfiguration           [] - Error 
> while trying to split key and value in configuration file /{working 
> dir}/flink-conf.yaml:{line num} "SELECT "
> ...{code}
> I dig into the source code and find out that the reader and writer methods of 
> YAML file cannot properly parse YAML format. Maybe it would be better to use 
> some third-party tools, such as snakeyaml, to do this job.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to