[jira] [Updated] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown

2018-04-26 Thread yuqi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLINK-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yuqi updated FLINK-9196:

Attachment: 0001-xxx.patch

> YARN: Flink binaries are not deleted from HDFS after cluster shutdown
> -
>
> Key: FLINK-9196
> URL: https://issues.apache.org/jira/browse/FLINK-9196
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
> Attachments: 0001-xxx.patch
>
>
> When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
> HDFS after the cluster shuts down.
> *Steps to reproduce*
> # Submit job in YARN job mode, non-detached:
> {noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
> -yjm 2048 -ytm 2048 ./examples/streaming/WordCount.jar {noformat}
> # Check contents of {{/user/hadoop/.flink/}} on HDFS after 
> job is finished:
> {noformat}
> [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
> /user/hadoop/.flink/application_1523966184826_0016
> Found 6 items
> -rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
> -rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
> -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
> /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
> drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/lib
> -rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
> -rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/logback.xml
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown

2018-04-26 Thread Gary Yao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLINK-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao updated FLINK-9196:

Description: 
When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
HDFS after the cluster shuts down.

*Steps to reproduce*
# Submit job in YARN job mode, non-detached:
{noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
-yjm 2048 -ytm 2048 ./examples/streaming/WordCount.jar {noformat}
# Check contents of {{/user/hadoop/.flink/}} on HDFS after job 
is finished:
{noformat}
[hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
/user/hadoop/.flink/application_1523966184826_0016
Found 6 items
-rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
-rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
-rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
/user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/lib
-rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/log4j.properties
-rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/logback.xml
{noformat}


  was:
When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
HDFS after the cluster shuts down.

*Steps to reproduce*
# Submit job in YARN job mode, non-detached:
{noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
-yjm 2048 -ytm 2048  ./examples/streaming/WordCount.jar {noformat}
# Check contents of {{/user/hadoop/.flink/}} on HDFS after job 
is finished:
{noformat}
[hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
/user/hadoop/.flink/application_1523966184826_0016
Found 6 items
-rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
-rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
-rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
/user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/lib
-rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/log4j.properties
-rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/logback.xml
{noformat}



> YARN: Flink binaries are not deleted from HDFS after cluster shutdown
> -
>
> Key: FLINK-9196
> URL: https://issues.apache.org/jira/browse/FLINK-9196
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
> HDFS after the cluster shuts down.
> *Steps to reproduce*
> # Submit job in YARN job mode, non-detached:
> {noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
> -yjm 2048 -ytm 2048 ./examples/streaming/WordCount.jar {noformat}
> # Check contents of {{/user/hadoop/.flink/}} on HDFS after 
> job is finished:
> {noformat}
> [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
> /user/hadoop/.flink/application_1523966184826_0016
> Found 6 items
> -rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
> -rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
> -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
> /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
> drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/lib
> -rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
> -rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/logback.xml
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown

2018-04-26 Thread Gary Yao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLINK-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao updated FLINK-9196:

Description: 
When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
HDFS after the cluster shuts down.

*Steps to reproduce*
# Submit job in YARN job mode, non-detached:
{noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
-yjm 2048 -ytm 2048  ./examples/streaming/WordCount.jar {noformat}
# Check contents of {{/user/hadoop/.flink/}} on HDFS after job 
is finished:
{noformat}
[hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
/user/hadoop/.flink/application_1523966184826_0016
Found 6 items
-rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
-rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
-rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
/user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/lib
-rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/log4j.properties
-rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/logback.xml
{noformat}


  was:
When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
HDFS after the cluster shuts down.

*Steps to reproduce*
# Submit job in YARN job mode, non-detached:
{noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
-yjm 2048 -ytm 2048 -yn 2  ./examples/streaming/WordCount.jar {noformat}
# Check contents of {{/user/hadoop/.flink/}} on HDFS after job 
is finished:
{noformat}
[hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
/user/hadoop/.flink/application_1523966184826_0016
Found 6 items
-rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
-rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
-rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
/user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
/user/hadoop/.flink/application_1523966184826_0016/lib
-rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/log4j.properties
-rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
/user/hadoop/.flink/application_1523966184826_0016/logback.xml
{noformat}



> YARN: Flink binaries are not deleted from HDFS after cluster shutdown
> -
>
> Key: FLINK-9196
> URL: https://issues.apache.org/jira/browse/FLINK-9196
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
> HDFS after the cluster shuts down.
> *Steps to reproduce*
> # Submit job in YARN job mode, non-detached:
> {noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
> -yjm 2048 -ytm 2048  ./examples/streaming/WordCount.jar {noformat}
> # Check contents of {{/user/hadoop/.flink/}} on HDFS after 
> job is finished:
> {noformat}
> [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
> /user/hadoop/.flink/application_1523966184826_0016
> Found 6 items
> -rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
> -rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
> -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
> /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
> drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/lib
> -rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
> -rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/logback.xml
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3

[jira] [Updated] (FLINK-9196) YARN: Flink binaries are not deleted from HDFS after cluster shutdown

2018-04-18 Thread Aljoscha Krettek (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLINK-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-9196:

Fix Version/s: 1.5.0

> YARN: Flink binaries are not deleted from HDFS after cluster shutdown
> -
>
> Key: FLINK-9196
> URL: https://issues.apache.org/jira/browse/FLINK-9196
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Affects Versions: 1.5.0
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> When deploying on YARN in flip6 mode, the Flink binaries are not deleted from 
> HDFS after the cluster shuts down.
> *Steps to reproduce*
> # Submit job in YARN job mode, non-detached:
> {noformat} HADOOP_CLASSPATH=`hadoop classpath` bin/flink run -m yarn-cluster 
> -yjm 2048 -ytm 2048 -yn 2  ./examples/streaming/WordCount.jar {noformat}
> # Check contents of {{/user/hadoop/.flink/}} on HDFS after 
> job is finished:
> {noformat}
> [hadoop@ip-172-31-43-78 flink-1.5.0]$ hdfs dfs -ls 
> /user/hadoop/.flink/application_1523966184826_0016
> Found 6 items
> -rw-r--r--   1 hadoop hadoop583 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/90cf5b3a-039e-4d52-8266-4e9563d74827-taskmanager-conf.yaml
> -rw-r--r--   1 hadoop hadoop332 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/application_1523966184826_0016-flink-conf.yaml3818971235442577934.tmp
> -rw-r--r--   1 hadoop hadoop   89779342 2018-04-02 17:08 
> /user/hadoop/.flink/application_1523966184826_0016/flink-dist_2.11-1.5.0.jar
> drwxrwxrwx   - hadoop hadoop  0 2018-04-17 14:54 
> /user/hadoop/.flink/application_1523966184826_0016/lib
> -rw-r--r--   1 hadoop hadoop   1939 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/log4j.properties
> -rw-r--r--   1 hadoop hadoop   2331 2018-04-02 15:37 
> /user/hadoop/.flink/application_1523966184826_0016/logback.xml
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)